09/01/2025 01:34:54.874 7f4910dbae00 [contrail-kube-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 01:34:54.874 7f4910dbae00 [contrail-kube-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 01:34:54.874 7f4910dbae00 [contrail-kube-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 01:34:54.875 7f490df05860 [contrail-kube-manager] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 01:34:54.875 7f490df05860 [contrail-kube-manager] [DEBUG]: Add uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:34:54.875 7f490df05860 [contrail-kube-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 01:34:54.875 7f490df05860 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 01:34:54.875 7f490df05860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.876 7f490df05860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:54.876 7f490df05860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.876 7f490df05860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.876 7f490df05860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 start_time = 1756690494875696 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/01/2025 01:34:54.876 7f490df05860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.920 7f4910dbae00 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 09/01/2025 01:34:54.928 7f4910dbae00 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:34:54.928 7f490d1e0f40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.929 7f490d1e0f40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:34:54.929 7f490d1e0f40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.929 7f490d1e0f40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.929 7f490d1e0f40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:34:54.929 7f490d1e0f40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.931 7f4910dbae00 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:34:54.932 7f490d1f62c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.932 7f490d1f62c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:54.932 7f490d1f62c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.940 7f4910dbae00 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:34:54.940 7f490d1ab720 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.940 7f490d1ab720 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:34:54.941 7f490d1ab720 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.947 7f4910dbae00 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:34:54.948 7f490d1ab720 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.948 7f490d1ab720 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:54.948 7f490d1ab720 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.961 7f4910dbae00 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Removed remained AMQP queue 09/01/2025 01:34:54.961 7f490d1d8a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.961 7f490d1d8a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 01:34:54.962 7f490d1d8a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.969 7f4910dbae00 [contrail-kube-manager] [WARNING]: Waiting to be elected as master... 09/01/2025 01:34:54.969 7f4910dbae00 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master... 09/01/2025 01:34:54.970 7f490d1d8a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:54.970 7f490d1d8a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:54.970 7f490d1d8a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.113 7f4910dbae00 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing... 09/01/2025 01:34:55.113 7f4910dbae00 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing... 09/01/2025 01:34:55.113 7f4910dbae00 [contrail-kube-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8108 09/01/2025 01:34:55.115 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add [config_updater.xml:/usr/local/lib/python3.9/site-packages/cfgm_common/uve/config_updater] to home page 09/01/2025 01:34:55.115 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 09/01/2025 01:34:55.116 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add [greenlets.xml:/usr/local/lib/python3.9/site-packages/cfgm_common/uve/greenlets] to home page 09/01/2025 01:34:55.116 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 09/01/2025 01:34:55.117 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add [cpuinfo.xml:/usr/local/lib/python3.9/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 09/01/2025 01:34:55.117 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add [kube_introspect.xml:/usr/local/lib/python3.9/site-packages/kube_manager/sandesh/kube_introspect] to home page 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_PodDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_NamespaceDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_ServiceDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_NetworkPolicyDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_IngressDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_LoadbalancerDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_LoadbalancerListenerDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_LoadbalancerPoolDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_LoadbalancerMemberDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_HealthMonitorDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_VirtualMachineDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_VirtualRouterDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_InstanceIpDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_ProjectDatabaseList] 09/01/2025 01:34:55.121 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_DomainDatabaseList] 09/01/2025 01:34:55.122 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SecurityGroupDatabaseList] 09/01/2025 01:34:55.122 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_FloatingIpPoolDatabaseList] 09/01/2025 01:34:55.122 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_FloatingIpDatabaseList] 09/01/2025 01:34:55.122 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_NetworkIpamDatabaseList] 09/01/2025 01:34:55.122 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_KubeGreenletStackList] 09/01/2025 01:34:55.122 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_KubeApiConnectionStatus] 09/01/2025 01:34:55.122 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_MastershipStatus] 09/01/2025 01:34:55.124 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/local/lib/python3.9/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 09/01/2025 01:34:55.124 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 09/01/2025 01:34:55.124 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 09/01/2025 01:34:55.124 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/local/lib/python3.9/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/local/lib/python3.9/site-packages/pysandesh/gen_py/sandesh_trace] to home page 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 09/01/2025 01:34:55.125 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 09/01/2025 01:34:55.126 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/local/lib/python3.9/site-packages/pysandesh/gen_py/sandesh_uve] to home page 09/01/2025 01:34:55.126 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 09/01/2025 01:34:55.126 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 09/01/2025 01:34:55.126 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 09/01/2025 01:34:55.127 7f4910dbae00 [contrail-kube-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 09/01/2025 01:34:55.128 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: KubeNetworkManager - init start 09/01/2025 01:34:55.136 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: KubeMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:55.137 7f490d1240e0 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port 09/01/2025 01:34:55.137 7f490d1240e0 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108 09/01/2025 01:34:55.137 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: KubeMonitor - Connect Kube API result 0 09/01/2025 01:34:55.218 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: KubeMonitor - KubeMonitor init done: url=https://10.0.0.50:6443/api/v1 09/01/2025 01:34:55.225 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - KubeMonitor init done: url=https://10.0.0.50:6443/api/v1 09/01/2025 01:34:55.231 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - KubeMonitor init done: url=https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1 09/01/2025 01:34:55.237 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - KubeMonitor init done: url=https://10.0.0.50:6443/api/v1 09/01/2025 01:34:55.242 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - KubeMonitor init done: url=https://10.0.0.50:6443/api/v1 09/01/2025 01:34:55.248 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - KubeMonitor init done: url=https://10.0.0.50:6443/apis/networking.k8s.io/v1 09/01/2025 01:34:55.254 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - KubeMonitor init done: url=https://10.0.0.50:6443/api/v1 09/01/2025 01:34:55.260 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - KubeMonitor init done: url=https://10.0.0.50:6443/apis/networking.k8s.io/v1 09/01/2025 01:34:55.260 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: VncKubernetes - vnc_connect starting 09/01/2025 01:34:55.264 7f490e2bacc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.264 7f490e2bacc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 09/01/2025 01:34:55.264 7f490e2bacc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.268 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: VncKubernetes - vnc_connect done 09/01/2025 01:34:55.268 7f4910dbae00 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql 09/01/2025 01:34:55.268 7f4910dbae00 [contrail-kube-manager] [WARNING]: initializing cassandra cluster 09/01/2025 01:34:55.269 7f4910dbae00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/01/2025 01:34:55.269 7f490e2bacc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.269 7f490e2bacc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:55.269 7f490e2bacc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.269 7f490e2bacc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.269 7f490e2bacc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:34:55.269 7f490e2bacc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.271 7f4910dbae00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/01/2025 01:34:55.289 7f4910dbae00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 01:34:55.290 7f490cf750e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.290 7f490cf750e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:34:55.290 7f490cf750e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.293 7f4910dbae00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 01:34:55.294 7f490cf75180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.294 7f490cf75180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 01:34:55.294 7f490cf75180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.297 7f4910dbae00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 01:34:55.304 7f4910dbae00 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 01:34:55.305 7f490cfa6ae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.305 7f490cfa6ae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:55.305 7f490cfa6ae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.318 7f4910dbae00 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.50:5673//'] 09/01/2025 01:34:55.318 7f4910dbae00 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 09/01/2025 01:34:55.318 7f490e2baae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.319 7f490e2baae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:34:55.319 7f490e2baae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.319 7f490e2baae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.319 7f490e2baae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Down description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:34:55.319 7f490e2baae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.327 7f4910dbae00 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 01:34:55.327 7f490e2baae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.328 7f490e2baae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:55.328 7f490e2baae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.334 7f4910dbae00 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:34:55.335 7f490e2baae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.335 7f490e2baae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Down description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:34:55.335 7f490e2baae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.337 7f4910dbae00 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:34:55.337 7f490e2baae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.337 7f490e2baae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:55.337 7f490e2baae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.338 7f490e2baae0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 09/01/2025 01:34:55.338 7f490cfa60e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.338 7f490cfa60e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Down description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 01:34:55.339 7f490cfa60e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.341 7f490e2baae0 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 01:34:55.342 7f490d1e0f40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.342 7f490d1e0f40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 01:34:55.342 7f490d1e0f40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:55.377 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: default-domain domain available. 09/01/2025 01:34:55.397 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default'], 'obj_dict': {'display_name': 'k8s-default', 'fq_name': ['default-domain', 'k8s-default'], 'id_perms': {'created': '2025-09-01T01:34:55.387855', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:55.387855', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11357663924834040824, 'uuid_mslong': 10386380444838743678}}, 'parent_type': 'domain', 'parent_uuid': 'ce056a4b-12ea-41a5-896d-393d506f2011', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'}, 'oper': 'CREATE', 'request-id': 'req-bd191f7e-5bc6-4e0b-b0dc-2d406720360c', 'type': 'project', 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'} 09/01/2025 01:34:55.426 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'display_name': 'default-application-policy-set', 'fq_name': ['default-domain', 'k8s-default', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:34:55.418880', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:55.418880', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12899231214931829714, 'uuid_mslong': 17526535842386496605}}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'f33ac37d-3499-4c5d-b303-467269a463d2'}, 'oper': 'CREATE', 'request-id': 'req-bd191f7e-5bc6-4e0b-b0dc-2d406720360c', 'type': 'application_policy_set', 'uuid': 'f33ac37d-3499-4c5d-b303-467269a463d2'} 09/01/2025 01:34:55.447 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default'], 'oper': 'UPDATE', 'request-id': 'req-bd191f7e-5bc6-4e0b-b0dc-2d406720360c', 'type': 'project', 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'} 09/01/2025 01:34:55.533 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'default'], 'obj_dict': {'display_name': 'default', 'fq_name': ['default-domain', 'k8s-default', 'default'], 'id_perms': {'created': '2025-09-01T01:34:55.484361', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:34:55.484361', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11936403218885516667, 'uuid_mslong': 6540726343857162531}}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': [{'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'a9777858-b10b-4f90-ae9a-ee86bb4e8bb1', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-default:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'a0ce129a-72d1-4990-8cd9-2cadac31ddea', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-default:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'f588a086-f096-477b-bc3b-6a0e980ef7ed', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '332d697a-4608-42c5-bfb1-5aa301238183', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}]}, 'security_group_id': 8000002, 'uuid': '5ac553f4-9aa5-4923-a5a6-9f7c5d801d7b'}, 'oper': 'CREATE', 'request-id': 'req-87bead24-6f46-492a-b835-8fdf789d0ce8', 'type': 'security_group', 'uuid': '5ac553f4-9aa5-4923-a5a6-9f7c5d801d7b'} 09/01/2025 01:34:55.617 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'default', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000002', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'a9777858-b10b-4f90-ae9a-ee86bb4e8bb1'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000002', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'a0ce129a-72d1-4990-8cd9-2cadac31ddea'}], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-default', 'default', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:34:55.565399', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:55.565399', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12393600091174732163, 'uuid_mslong': 5435922950334663122}}, 'parent_type': 'security-group', 'parent_uuid': '5ac553f4-9aa5-4923-a5a6-9f7c5d801d7b', 'perms2': {'global_access': 0, 'owner': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'owner_access': 7, 'share': []}, 'uuid': '4b70471f-6117-49d2-abfe-e99e6ab80183'}, 'oper': 'CREATE', 'request-id': 'req-1aaa4255-fd97-4aa1-8586-a82862deafbe', 'type': 'access_control_list', 'uuid': '4b70471f-6117-49d2-abfe-e99e6ab80183'} 09/01/2025 01:34:55.658 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg'], 'obj_dict': {'display_name': 'k8s-default-default-sg', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg'], 'id_perms': {'created': '2025-09-01T01:34:55.649202', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:34:55.649202', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13595223614054563807, 'uuid_mslong': 5613512272088154649}}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': []}, 'security_group_id': 8000003, 'uuid': '4de733ab-781c-4219-bcab-efd8adb97fdf'}, 'oper': 'CREATE', 'request-id': 'req-862169ea-8f04-4eb4-b8a9-323bb8eb9c3a', 'type': 'security_group', 'uuid': '4de733ab-781c-4219-bcab-efd8adb97fdf'} 09/01/2025 01:34:55.731 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'default', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'f588a086-f096-477b-bc3b-6a0e980ef7ed'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '332d697a-4608-42c5-bfb1-5aa301238183'}], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-default', 'default', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:34:55.667490', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:55.667490', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13388604008182575585, 'uuid_mslong': 7374738075550172945}}, 'parent_type': 'security-group', 'parent_uuid': '5ac553f4-9aa5-4923-a5a6-9f7c5d801d7b', 'perms2': {'global_access': 0, 'owner': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'owner_access': 7, 'share': []}, 'uuid': '66585534-e8b7-4711-b9cd-e068da17f1e1'}, 'oper': 'CREATE', 'request-id': 'req-dc2a2ba5-0558-47fa-ae44-cc96394be8fc', 'type': 'access_control_list', 'uuid': '66585534-e8b7-4711-b9cd-e068da17f1e1'} 09/01/2025 01:34:55.742 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-8e1185cb-738d-413f-9236-17f30c42c65b', 'type': 'security_group', 'uuid': '4de733ab-781c-4219-bcab-efd8adb97fdf'} 09/01/2025 01:34:55.823 7f490cfa6680 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:34:55.828 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:34:55.820914', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:55.820914', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10221026790512302696, 'uuid_mslong': 724960809861532916}}, 'parent_type': 'security-group', 'parent_uuid': '4de733ab-781c-4219-bcab-efd8adb97fdf', 'perms2': {'global_access': 0, 'owner': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'owner_access': 7, 'share': []}, 'uuid': '0a0f9403-fbe0-4cf4-8dd8-61b6703c9a68'}, 'oper': 'CREATE', 'request-id': 'req-54de4dcc-5324-4370-98b0-d85f92399627', 'type': 'access_control_list', 'uuid': '0a0f9403-fbe0-4cf4-8dd8-61b6703c9a68'} 09/01/2025 01:34:55.843 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'obj_dict': {'display_name': 'k8s', 'fq_name': ['default-policy-management', 'k8s'], 'id_perms': {'created': '2025-09-01T01:34:55.834723', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:55.834723', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12603312532585330301, 'uuid_mslong': 1917128780955733221}}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'}, 'oper': 'CREATE', 'request-id': 'req-a0620321-560d-4dab-b586-65ca6b69f87f', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:55.868 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:34:55.857136', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:55.857136', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10774130541549910497, 'uuid_mslong': 16845570577868801345}}, 'parent_type': 'security-group', 'parent_uuid': '4de733ab-781c-4219-bcab-efd8adb97fdf', 'perms2': {'global_access': 0, 'owner': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'owner_access': 7, 'share': []}, 'uuid': 'e9c77d30-5ca4-4541-9585-66a4ccc4f9e1'}, 'oper': 'CREATE', 'request-id': 'req-7f692d85-96b4-4f98-8275-a07a203ce93c', 'type': 'access_control_list', 'uuid': 'e9c77d30-5ca4-4541-9585-66a4ccc4f9e1'} 09/01/2025 01:34:55.918 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-eef11707-29e3-4fad-9cd7-903ad0babe99', 'type': 'global_vrouter_config', 'uuid': '87a687b0-9781-4994-aa87-dbc3f38997c0'} 09/01/2025 01:34:55.991 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'obj_dict': {'display_name': 'k8s-kube-system', 'fq_name': ['default-domain', 'k8s-kube-system'], 'id_perms': {'created': '2025-09-01T01:34:55.985402', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:55.985402', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10013454790172241632, 'uuid_mslong': 11951427749879825585}}, 'parent_type': 'domain', 'parent_uuid': 'ce056a4b-12ea-41a5-896d-393d506f2011', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'}, 'oper': 'CREATE', 'request-id': 'req-41dd65dc-f795-45e5-a745-6d1962e85f36', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:34:56.016 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'display_name': 'default-application-policy-set', 'fq_name': ['default-domain', 'k8s-kube-system', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:34:56.012086', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.012086', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10948140934525109059, 'uuid_mslong': 17021587060483244415}}, 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'ec38d33f-4d7d-417f-97ef-9c2c99484343'}, 'oper': 'CREATE', 'request-id': 'req-41dd65dc-f795-45e5-a745-6d1962e85f36', 'type': 'application_policy_set', 'uuid': 'ec38d33f-4d7d-417f-97ef-9c2c99484343'} 09/01/2025 01:34:56.044 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-41dd65dc-f795-45e5-a745-6d1962e85f36', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:34:56.295 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'default'], 'obj_dict': {'display_name': 'default', 'fq_name': ['default-domain', 'k8s-kube-system', 'default'], 'id_perms': {'created': '2025-09-01T01:34:56.235704', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:34:56.235704', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11024785080048304205, 'uuid_mslong': 14009118518343257964}}, 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': [{'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '0343e44d-b0e1-4539-8c19-d2156da9cc62', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-kube-system:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'a92a8a05-b289-4b87-89c3-98c0865faf30', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-kube-system:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '9cd8ca77-a919-448f-88f9-6db9c556a8f1', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '5f74e63a-1bb1-40ad-9d17-b59bdd9d5719', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}]}, 'security_group_id': 8000004, 'uuid': 'c26a6311-a632-4f6c-98ff-e79e554e404d'}, 'oper': 'CREATE', 'request-id': 'req-4a0481de-ef18-4672-b4dd-de9aaa349670', 'type': 'security_group', 'uuid': 'c26a6311-a632-4f6c-98ff-e79e554e404d'} 09/01/2025 01:34:56.352 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg'], 'obj_dict': {'display_name': 'k8s-kube-system-default-sg', 'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg'], 'id_perms': {'created': '2025-09-01T01:34:56.343303', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:34:56.343303', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11198606753506992789, 'uuid_mslong': 16211337997743114230}}, 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': []}, 'security_group_id': 8000005, 'uuid': 'e0fa3e04-8145-47f6-9b69-718272132295'}, 'oper': 'CREATE', 'request-id': 'req-ccd58c30-7b81-4365-9b61-97e54a8c03ce', 'type': 'security_group', 'uuid': 'e0fa3e04-8145-47f6-9b69-718272132295'} 09/01/2025 01:34:56.419 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'default', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000004', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '0343e44d-b0e1-4539-8c19-d2156da9cc62'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000004', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'a92a8a05-b289-4b87-89c3-98c0865faf30'}], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-system', 'default', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:34:56.357752', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.357752', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13076623060756524689, 'uuid_mslong': 3924770634606397527}}, 'parent_type': 'security-group', 'parent_uuid': 'c26a6311-a632-4f6c-98ff-e79e554e404d', 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'uuid': '36779619-0c99-4857-b579-7f640e12ee91'}, 'oper': 'CREATE', 'request-id': 'req-88880579-f6ed-40c8-8e6d-21e041eb2eac', 'type': 'access_control_list', 'uuid': '36779619-0c99-4857-b579-7f640e12ee91'} 09/01/2025 01:34:56.436 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-240070e3-47f5-4369-b66e-571fc8a75cef', 'type': 'security_group', 'uuid': 'e0fa3e04-8145-47f6-9b69-718272132295'} 09/01/2025 01:34:56.513 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'default', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '9cd8ca77-a919-448f-88f9-6db9c556a8f1'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '5f74e63a-1bb1-40ad-9d17-b59bdd9d5719'}], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-system', 'default', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:34:56.457597', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.457597', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11506460743523033639, 'uuid_mslong': 6215132166285246947}}, 'parent_type': 'security-group', 'parent_uuid': 'c26a6311-a632-4f6c-98ff-e79e554e404d', 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'uuid': '564095c6-ab89-41e3-9faf-291517aae227'}, 'oper': 'CREATE', 'request-id': 'req-ac8f95ef-cfd2-4c90-b51d-d56b104064e6', 'type': 'access_control_list', 'uuid': '564095c6-ab89-41e3-9faf-291517aae227'} 09/01/2025 01:34:56.526 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-ip-fabric-ipam'], 'obj_dict': {'display_name': 'k8s-ip-fabric-ipam', 'fq_name': ['default-domain', 'k8s-default', 'k8s-ip-fabric-ipam'], 'id_perms': {'created': '2025-09-01T01:34:56.516615', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.516615', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12876161066528670497, 'uuid_mslong': 14866473943478387208}}, 'ipam_subnet_method': 'flat-subnet', 'ipam_subnets': {'subnets': [{'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': '10.79.255.254', 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': '10.79.255.253', 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '10.64.0.0', 'ip_prefix_len': 12}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}]}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'ce505349-1fa1-4208-b2b1-5044e9280321'}, 'oper': 'CREATE', 'request-id': 'req-91f7aff8-08b6-4a47-bbf5-4166f8477cc5', 'type': 'network_ipam', 'uuid': 'ce505349-1fa1-4208-b2b1-5044e9280321'} 09/01/2025 01:34:56.565 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-pod-ipam'], 'obj_dict': {'display_name': 'k8s-pod-ipam', 'fq_name': ['default-domain', 'k8s-default', 'k8s-pod-ipam'], 'id_perms': {'created': '2025-09-01T01:34:56.554258', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.554258', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10520340932493082396, 'uuid_mslong': 5725016635900381083}}, 'ipam_subnet_method': 'flat-subnet', 'ipam_subnets': {'subnets': [{'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': '10.47.255.254', 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': '10.47.255.253', 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '10.32.0.0', 'ip_prefix_len': 12}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}]}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '4f73584c-440b-479b-91ff-c256c5407f1c'}, 'oper': 'CREATE', 'request-id': 'req-407f4093-370d-4446-804c-7616fae2fc43', 'type': 'network_ipam', 'uuid': '4f73584c-440b-479b-91ff-c256c5407f1c'} 09/01/2025 01:34:56.587 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:34:56.581158', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.581158', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9762737177291075290, 'uuid_mslong': 2998260824958910946}}, 'parent_type': 'security-group', 'parent_uuid': 'e0fa3e04-8145-47f6-9b69-718272132295', 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'uuid': '299bf655-b7a5-41e2-877c-35bccfef2eda'}, 'oper': 'CREATE', 'request-id': 'req-9f39a7cb-d7d5-4f96-bf25-7339bd7a5d2f', 'type': 'access_control_list', 'uuid': '299bf655-b7a5-41e2-877c-35bccfef2eda'} 09/01/2025 01:34:56.700 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'obj_dict': {'address_allocation_mode': 'flat-subnet-only', 'display_name': 'k8s-default-pod-network', 'fabric_snat': True, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'id_perms': {'created': '2025-09-01T01:34:56.642429', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.642429', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10518379392776784806, 'uuid_mslong': 15443089546972253873}}, 'is_shared': False, 'network_ipam_refs': [{'attr': {'host_routes': None, 'ipam_subnets': [{'subnet_uuid': 'eada5f2d-448e-4621-9fea-812925eb46e9'}]}, 'to': ['default-domain', 'k8s-default', 'k8s-pod-ipam'], 'uuid': '4f73584c-440b-479b-91ff-c256c5407f1c'}], 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6', 'virtual_network_network_id': 7, 'virtual_network_properties': {'allow_transit': None, 'forwarding_mode': 'l3', 'max_flows': 0, 'mirror_destination': False, 'mtu': None, 'network_id': None, 'rpf': None, 'vxlan_network_identifier': None}}, 'oper': 'CREATE', 'request-id': 'req-59aee991-1a94-429a-b425-b319a40967e8', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:34:56.718 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:34:56.692301', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.692301', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11196397370808665147, 'uuid_mslong': 4939850386392500102}}, 'parent_type': 'security-group', 'parent_uuid': 'e0fa3e04-8145-47f6-9b69-718272132295', 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'uuid': '448ddfc1-0480-4786-9b61-9816732b043b'}, 'oper': 'CREATE', 'request-id': 'req-226d9218-d6e4-44e4-8425-5c01a5bf858b', 'type': 'access_control_list', 'uuid': '448ddfc1-0480-4786-9b61-9816732b043b'} 09/01/2025 01:34:56.756 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'obj_dict': {'display_name': 'k8s-default-pod-network', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'id_perms': {'created': '2025-09-01T01:34:56.741593', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.741593', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9857280386579676206, 'uuid_mslong': 6892182549909753541}}, 'parent_type': 'virtual-network', 'parent_uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': True, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'}, 'oper': 'CREATE', 'request-id': 'req-59aee991-1a94-429a-b425-b319a40967e8', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:34:56.860 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-pod-network'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-pod-network'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-pod-network'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'deny'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-pod-network'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000002'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'k8s-default-pod-network', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'id_perms': {'created': '2025-09-01T01:34:56.808303', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.808303', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12634401716669227668, 'uuid_mslong': 9014006186852894039}}, 'parent_type': 'virtual-network', 'parent_uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '7d182ede-628a-4557-af56-696bdfa7a294'}, 'oper': 'CREATE', 'request-id': 'req-1c8d35d4-fa79-4b1e-be60-502b66b7db78', 'type': 'access_control_list', 'uuid': '7d182ede-628a-4557-af56-696bdfa7a294'} 09/01/2025 01:34:56.899 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['target:64512:8000004'], 'obj_dict': {'display_name': 'target:64512:8000004', 'fq_name': ['target:64512:8000004'], 'id_perms': {'created': '2025-09-01T01:34:56.891155', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.891155', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13143918234955476838, 'uuid_mslong': 13547113033425112061}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'bc010387-395e-4bfd-b668-93fec74cef66'}, 'oper': 'CREATE', 'request-id': 'req-32a5bfe9-d020-47b3-8268-e0dcf80f619e', 'type': 'route_target', 'uuid': 'bc010387-395e-4bfd-b668-93fec74cef66'} 09/01/2025 01:34:56.919 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-service-ipam'], 'obj_dict': {'display_name': 'k8s-service-ipam', 'fq_name': ['default-domain', 'k8s-default', 'k8s-service-ipam'], 'id_perms': {'created': '2025-09-01T01:34:56.901828', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:56.901828', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10632512379082133835, 'uuid_mslong': 17731803175260668569}}, 'ipam_subnet_method': 'flat-subnet', 'ipam_subnets': {'subnets': [{'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': '10.111.255.254', 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': '10.111.255.253', 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '10.96.0.0', 'ip_prefix_len': 12}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}]}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'f614050a-708e-4299-938e-45ace122514b'}, 'oper': 'CREATE', 'request-id': 'req-c80b4929-fcc0-4eb7-96d7-1f51333b6e1a', 'type': 'network_ipam', 'uuid': 'f614050a-708e-4299-938e-45ace122514b'} 09/01/2025 01:34:56.955 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-e7ae3a1d-9b1b-4bd4-be39-41c1c5a4cdc5', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:34:57.013 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-fec8e9ff-0f7e-4854-abee-9b44fc1073dc', 'type': 'access_control_list', 'uuid': '7d182ede-628a-4557-af56-696bdfa7a294'} 09/01/2025 01:34:57.097 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'obj_dict': {'address_allocation_mode': 'flat-subnet-only', 'display_name': 'k8s-default-service-network', 'fabric_snat': False, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'id_perms': {'created': '2025-09-01T01:34:57.045929', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:57.045929', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11820567914732569193, 'uuid_mslong': 12016753917161719144}}, 'is_shared': False, 'network_ipam_refs': [{'attr': {'host_routes': None, 'ipam_subnets': [{'subnet_uuid': 'cdf5a7bb-3fee-45f9-9299-6a9e2ef0e400'}]}, 'to': ['default-domain', 'k8s-default', 'k8s-service-ipam'], 'uuid': 'f614050a-708e-4299-938e-45ace122514b'}], 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269', 'virtual_network_network_id': 8, 'virtual_network_properties': {'allow_transit': None, 'forwarding_mode': 'l3', 'max_flows': 0, 'mirror_destination': False, 'mtu': None, 'network_id': None, 'rpf': None, 'vxlan_network_identifier': None}}, 'oper': 'CREATE', 'request-id': 'req-232d5b26-5459-4cd8-acd5-13056167b54e', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:34:57.142 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'obj_dict': {'display_name': 'k8s-default-service-network', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'id_perms': {'created': '2025-09-01T01:34:57.129557', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:57.129557', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10058874526549187742, 'uuid_mslong': 11572430850722186631}}, 'parent_type': 'virtual-network', 'parent_uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'}, 'oper': 'CREATE', 'request-id': 'req-232d5b26-5459-4cd8-acd5-13056167b54e', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:34:57.178 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['target:64512:8000005'], 'obj_dict': {'display_name': 'target:64512:8000005', 'fq_name': ['target:64512:8000005'], 'id_perms': {'created': '2025-09-01T01:34:57.170046', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:57.170046', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13218141680492412823, 'uuid_mslong': 12160149632363022162}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a8c187a9-d498-4752-b770-45d36088ff97'}, 'oper': 'CREATE', 'request-id': 'req-c62be96c-759a-46f4-a171-07ccd4bfb462', 'type': 'route_target', 'uuid': 'a8c187a9-d498-4752-b770-45d36088ff97'} 09/01/2025 01:34:57.195 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-565d0cba-c30e-48eb-aea3-a2632f8f2e7a', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:34:57.236 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-ip-fabric-np'], 'obj_dict': {'display_name': 'k8s-default-ip-fabric-np', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-ip-fabric-np'], 'id_perms': {'created': '2025-09-01T01:34:57.224637', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:57.224637', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9820500363855709707, 'uuid_mslong': 6871729834595271054}}, 'network_policy_entries': {'policy_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'application': [], 'created': None, 'description': None, 'direction': '<>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}], 'dst_ports': [{'end_port': -1, 'start_port': -1}], 'ethertype': None, 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '8f3cc2a6-3f21-40bd-9cf8-07332875b75a', 'src_addresses': [{'network_policy': 'default-domain:k8s-default:k8s-default-ip-fabric-np', 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': -1, 'start_port': -1}]}]}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '5f5d49e0-6781-418e-8849-6d0cb836c60b'}, 'oper': 'CREATE', 'request-id': 'req-e1542e63-440c-4bfd-bd1d-9ed52a8104eb', 'type': 'network_policy', 'uuid': '5f5d49e0-6781-418e-8849-6d0cb836c60b'} 09/01/2025 01:34:57.306 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-np'], 'obj_dict': {'display_name': 'k8s-default-service-np', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-np'], 'id_perms': {'created': '2025-09-01T01:34:57.294843', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:57.294843', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12937301099142653850, 'uuid_mslong': 11970738815292819317}}, 'network_policy_entries': {'policy_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'application': [], 'created': None, 'description': None, 'direction': '<>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-service-network'}], 'dst_ports': [{'end_port': -1, 'start_port': -1}], 'ethertype': None, 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '2cc90ab6-4fd1-4d50-8836-f841cc9e0762', 'src_addresses': [{'network_policy': 'default-domain:k8s-default:k8s-default-service-np', 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': -1, 'start_port': -1}]}]}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'a6209b87-6082-4775-b38a-86ce0eab279a'}, 'oper': 'CREATE', 'request-id': 'req-734b6617-5e04-4ff6-99ab-f28409d88552', 'type': 'network_policy', 'uuid': 'a6209b87-6082-4775-b38a-86ce0eab279a'} 09/01/2025 01:34:57.347 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-service-np'], 'obj_dict': {'display_name': 'k8s-default-pod-service-np', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-service-np'], 'id_perms': {'created': '2025-09-01T01:34:57.336300', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:57.336300', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9434725081410064776, 'uuid_mslong': 2241212454514476764}}, 'network_policy_entries': {'policy_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'application': [], 'created': None, 'description': None, 'direction': '<>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-service-network'}], 'dst_ports': [{'end_port': -1, 'start_port': -1}], 'ethertype': None, 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '629589cc-223a-407e-a160-df2b0b524fff', 'src_addresses': [{'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-pod-network'}], 'src_ports': [{'end_port': -1, 'start_port': -1}]}]}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '1f1a62da-5984-46dc-82ee-e07966ea5188'}, 'oper': 'CREATE', 'request-id': 'req-104165ae-0cd3-4d5b-9ae2-ca203fee1e4e', 'type': 'network_policy', 'uuid': '1f1a62da-5984-46dc-82ee-e07966ea5188'} 09/01/2025 01:34:57.410 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-21093259-5a40-4df3-bbb7-8edf25d90339', 'type': 'virtual_network', 'uuid': 'e07e6e04-4e93-4043-ab83-32b8f8854bbe'} 09/01/2025 01:34:57.452 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-b1969f07-11f0-4f5a-acfd-26d748871cb4', 'type': 'access_control_list', 'uuid': 'a87aabc0-ac59-4e06-a71b-5e1b9d1a40ce'} 09/01/2025 01:34:57.453 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-21093259-5a40-4df3-bbb7-8edf25d90339', 'type': 'routing_instance', 'uuid': '23054292-38bc-4b6f-b5fa-0f8021a7362f'} 09/01/2025 01:34:57.489 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-780f1e29-426f-4c8a-95bd-66a39eb92d4d', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:34:57.517 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-1b6a50f4-d11d-4775-8b07-24c851eab479', 'type': 'access_control_list', 'uuid': 'a87aabc0-ac59-4e06-a71b-5e1b9d1a40ce'} 09/01/2025 01:34:57.523 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-k8s-1454-1'], 'obj_dict': {'config_database_node_ip_address': '10.0.0.50', 'display_name': 'cn-jenkins-deploy-platform-juju-k8s-1454-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-k8s-1454-1'], 'id_perms': {'created': '2025-09-01T01:34:57.511328', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:57.511328', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10288526971317814616, 'uuid_mslong': 9619987740220280603}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '85810fe1-7f71-4f1b-8ec8-30c4fb6c8558'}, 'oper': 'CREATE', 'request-id': 'req-bc40be73-7cda-43db-bf51-5d88a64adeb5', 'type': 'config_database_node', 'uuid': '85810fe1-7f71-4f1b-8ec8-30c4fb6c8558'} 09/01/2025 01:34:57.545 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-780f1e29-426f-4c8a-95bd-66a39eb92d4d', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:34:57.615 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-63d9d2e6-ba1a-4d0d-8a27-1157f1443220', 'type': 'access_control_list', 'uuid': '7d182ede-628a-4557-af56-696bdfa7a294'} 09/01/2025 01:34:57.634 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-ed81ae90-4f21-4fc8-85dc-fb2d26a6f954', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:34:57.646 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-f01d4e77-4220-4d4d-b01b-775c7abb911e', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:34:57.648 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-f01d4e77-4220-4d4d-b01b-775c7abb911e', 'type': 'routing_instance', 'uuid': '23054292-38bc-4b6f-b5fa-0f8021a7362f'} 09/01/2025 01:34:57.693 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-ed81ae90-4f21-4fc8-85dc-fb2d26a6f954', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:34:57.702 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-37941629-f84d-418f-9a40-ad7b740bf2eb', 'type': 'access_control_list', 'uuid': '7d182ede-628a-4557-af56-696bdfa7a294'} 09/01/2025 01:34:57.772 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-7513ce5c-b4bf-4702-9bf3-443ac5b9bfb9', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:34:57.813 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-7513ce5c-b4bf-4702-9bf3-443ac5b9bfb9', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:34:57.814 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-93c437c6-6633-4b6e-a8ae-a1e511faedc6', 'type': 'access_control_list', 'uuid': 'a87aabc0-ac59-4e06-a71b-5e1b9d1a40ce'} 09/01/2025 01:34:57.853 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-66ce2b60-91ef-454f-92d1-0a96443b837e', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:34:57.859 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': -1, 'start_port': -1}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-service-network'}, 'src_port': {'end_port': -1, 'start_port': -1}}, 'rule_uuid': '8f3cc2a6-3f21-40bd-9cf8-07332875b75a'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-service-network'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:k8s-default:k8s-default-service-network'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'k8s-default-service-network', 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'id_perms': {'created': '2025-09-01T01:34:57.838228', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:57.838228', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9688595357425536257, 'uuid_mslong': 962755083897160870}}, 'parent_type': 'virtual-network', 'parent_uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '0d5c64a5-dd64-44a6-8674-ce252bd90d01'}, 'oper': 'CREATE', 'request-id': 'req-0a977f31-731a-4fd1-a5df-e760cf5106de', 'type': 'access_control_list', 'uuid': '0d5c64a5-dd64-44a6-8674-ce252bd90d01'} 09/01/2025 01:34:57.907 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-1d8b72b9-3623-4b84-9199-10245016ba49', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:34:57.909 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-1d8b72b9-3623-4b84-9199-10245016ba49', 'type': 'routing_instance', 'uuid': '23054292-38bc-4b6f-b5fa-0f8021a7362f'} 09/01/2025 01:34:57.923 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-66ce2b60-91ef-454f-92d1-0a96443b837e', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:34:57.952 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-852aa536-24e0-4a8c-b109-95a6f9a39b58', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:34:58.003 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-6395e580-30d8-495c-ac31-7bebb8e20dad', 'type': 'access_control_list', 'uuid': '0d5c64a5-dd64-44a6-8674-ce252bd90d01'} 09/01/2025 01:34:58.029 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-852aa536-24e0-4a8c-b109-95a6f9a39b58', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:34:58.040 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-5ea10595-094d-412f-bbae-a518ae5ea01c', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:34:58.044 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-5ea10595-094d-412f-bbae-a518ae5ea01c', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:34:58.129 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: VncNetworkPolicy init done. 09/01/2025 01:34:58.275 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-denyall'], 'obj_dict': {'display_name': 'k8s-denyall', 'fq_name': ['default-policy-management', 'k8s-denyall'], 'id_perms': {'created': '2025-09-01T01:34:58.269037', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:58.269037', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9281677744765653977, 'uuid_mslong': 16276300850529848719}}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e1e10963-5538-4d8f-80cf-24bb8067d3d9'}, 'oper': 'CREATE', 'request-id': 'req-013bb0d9-3a7e-4f51-9114-ec4159349c3c', 'type': 'firewall_policy', 'uuid': 'e1e10963-5538-4d8f-80cf-24bb8067d3d9'} 09/01/2025 01:34:58.337 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-denyall'], 'oper': 'UPDATE', 'request-id': 'req-51c82e2c-a312-42d1-b532-2aa30f0232b1', 'type': 'firewall_policy', 'uuid': 'e1e10963-5538-4d8f-80cf-24bb8067d3d9'} 09/01/2025 01:34:58.453 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-bccd7199-869c-49f9-bc42-a24d93d7a01e', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:58.618 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'obj_dict': {'display_name': 'k8s-allowall', 'fq_name': ['default-policy-management', 'k8s-allowall'], 'id_perms': {'created': '2025-09-01T01:34:58.612519', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:58.612519', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11791485297466611927, 'uuid_mslong': 12591360059846116675}}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'}, 'oper': 'CREATE', 'request-id': 'req-eacc1f31-9300-4f08-9293-9539003f9be2', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:34:58.694 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-1f44f737-0863-4947-b5c3-f31c406646c1', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:34:58.841 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-c6b60776-7032-491d-a74d-53aa89fe5a38', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:58.875 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-7b527ba4-12f9-4141-957d-e71ac80e0977', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:58.876 7f490d022cc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:34:58.876 7f490d022cc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:34:58.876 7f490d022cc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:34:58.876 7f490d022cc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:34:58.877 7f490d022cc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:34:58.914 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-7c6f2258-bcda-4d16-9dba-264fa9d38c4e', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:59.050 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Ingress'], 'obj_dict': {'display_name': 'k8s-Ingress', 'fq_name': ['default-policy-management', 'k8s-Ingress'], 'id_perms': {'created': '2025-09-01T01:34:59.040105', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:59.040105', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10188735552209959056, 'uuid_mslong': 7715603300617307372}}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6b135452-a920-44ec-8d65-a900a3de3890'}, 'oper': 'CREATE', 'request-id': 'req-c0edc76c-a3f8-449f-b64c-b3aad4442d98', 'type': 'firewall_policy', 'uuid': '6b135452-a920-44ec-8d65-a900a3de3890'} 09/01/2025 01:34:59.082 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Ingress'], 'oper': 'UPDATE', 'request-id': 'req-c7a474ad-d2ab-43c0-a434-7c283897e862', 'type': 'firewall_policy', 'uuid': '6b135452-a920-44ec-8d65-a900a3de3890'} 09/01/2025 01:34:59.270 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-5d95b5ce-064e-4fb8-a834-2b173c32b6cb', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:59.334 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-206b120c-3131-4ae7-8935-b0755f77ee9f', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:59.365 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-3444dad5-67b5-4b4c-8fcc-0d82e1938fe5', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:59.437 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-80f3d144-97b9-4630-ab6d-f6c12edc818a', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:59.529 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-95997700-60ae-48d9-9394-8ef1de58aca0', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:59.744 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['application=k8s'], 'obj_dict': {'display_name': 'application=k8s', 'fq_name': ['application=k8s'], 'id_perms': {'created': '2025-09-01T01:34:59.735253', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:59.735253', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11942659685020229637, 'uuid_mslong': 2992592746691382189}}, 'name': 'application=k8s', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x00010000', 'tag_type_name': 'application', 'tag_type_refs': [{'to': ['application'], 'uuid': 'd092075d-9887-4533-9bdd-86b77d6c3c87'}], 'tag_value': 'k8s', 'uuid': '2987d33f-67f3-4bad-a5bc-d9b567026005'}, 'oper': 'CREATE', 'request-id': 'req-74ba4f4e-fe97-4ec0-bfd1-f8040e55371b', 'type': 'tag', 'uuid': '2987d33f-67f3-4bad-a5bc-d9b567026005'} 09/01/2025 01:34:59.835 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: KubeNetworkManager - init done 09/01/2025 01:34:59.835 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: Starting all tasks 09/01/2025 01:34:59.835 7f4910dbae00 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: Wait for 8 started tasks 09/01/2025 01:34:59.835 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:34:59.835 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:34:59.836 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=False, resource_version=None 09/01/2025 01:34:59.836 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:59.836 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:34:59.836 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=False, resource_version=None 09/01/2025 01:34:59.836 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:59.836 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:34:59.836 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=False, resource_version=None 09/01/2025 01:34:59.836 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:59.837 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:34:59.837 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=False, resource_version=None 09/01/2025 01:34:59.837 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:59.837 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:34:59.837 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=False, resource_version=None 09/01/2025 01:34:59.837 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:59.837 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:34:59.837 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=False, resource_version=None 09/01/2025 01:34:59.837 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:59.837 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:34:59.837 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=False, resource_version=None 09/01/2025 01:34:59.838 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:59.838 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 01:34:59.838 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start init url=https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses resource_version=None 09/01/2025 01:34:59.842 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:34:59.842 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start init url=https://10.0.0.50:6443/api/v1/endpoints resource_version=None 09/01/2025 01:34:59.844 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:34:59.845 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start init url=https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies resource_version=None 09/01/2025 01:34:59.846 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 01:34:59.846 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start init url=https://10.0.0.50:6443/api/v1/services resource_version=None 09/01/2025 01:34:59.847 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 01:34:59.847 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start init url=https://10.0.0.50:6443/api/v1/pods resource_version=None 09/01/2025 01:34:59.848 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:34:59.849 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:34:59.850 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start init url=https://10.0.0.50:6443/api/v1/namespaces resource_version=None 09/01/2025 01:34:59.851 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s'], 'oper': 'UPDATE', 'request-id': 'req-4d9bbb9e-f9da-425a-a88c-ad9a8d083f2b', 'type': 'application_policy_set', 'uuid': '1a9b0278-098b-4ce5-aee7-f5f921fa367d'} 09/01/2025 01:34:59.870 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Done init url=https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses, resource_version=381 09/01/2025 01:34:59.870 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:34:59.870 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:34:59.872 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:34:59.872 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:34:59.873 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:34:59.873 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:34:59.875 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Done init url=https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies, resource_version=381 09/01/2025 01:34:59.875 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:34:59.875 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:34:59.876 7f490d022900 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: NetworkMonitor - Retrieved following CRD info = {} 09/01/2025 01:34:59.877 7f490d022900 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: NetworkMonitor - Creating Network CRD 09/01/2025 01:34:59.878 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:34:59.878 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:34:59.879 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:34:59.882 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Got ADDED Pod kube-system:coredns-6f867cd986-wlkqx:a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8 09/01/2025 01:34:59.882 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Done init url=https://10.0.0.50:6443/api/v1/pods, resource_version=381 09/01/2025 01:34:59.882 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:34:59.882 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:34:59.883 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Got ADDED Namespace None:contrail:a28020d1-63f5-47e0-90e8-7a75e7378bf9 09/01/2025 01:34:59.884 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:34:59.884 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=3 timeout=120) 09/01/2025 01:34:59.884 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=coredns-6f867cd986-wlkqx event_type=ADDED kind=Pod ns=kube-system) 09/01/2025 01:34:59.885 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncPod - Got ADDED Pod kube-system:coredns-6f867cd986-wlkqx:a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8 09/01/2025 01:34:59.886 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Got ADDED Endpoints default:kubernetes:578b072c-0e11-408a-a97f-e0b0b4c830b3 09/01/2025 01:34:59.887 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:34:59.887 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Got ADDED Service default:kubernetes:4149e3a7-961c-4071-90b8-b63acefd96b1 09/01/2025 01:34:59.889 7f490d022900 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: NetworkMonitor - Creating Network CRD response - {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"network-attachment-definitions.k8s.cni.cncf.io","uid":"060b80da-99e6-4517-b783-a585521a3460","resourceVersion":"382","generation":1,"creationTimestamp":"2025-09-01T01:34:59Z","managedFields":[{"manager":"python-requests","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2025-09-01T01:34:59Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:conversion":{".":{},"f:strategy":{}},"f:group":{},"f:names":{"f:kind":{},"f:listKind":{},"f:plural":{},"f:shortNames":{},"f:singular":{}},"f:scope":{},"f:versions":{}}}}]},"spec":{"group":"k8s.cni.cncf.io","names":{"plural":"network-attachment-definitions","singular":"network-attachment-definition","shortNames":["net-attach-def"],"kind":"NetworkAttachmentDefinition","listKind":"NetworkAttachmentDefinitionList"},"scope":"Namespaced","versions":[{"name":"v1","served":true,"storage":true,"schema":{"openAPIV3Schema":{"type":"object","properties":{"spec":{"type":"object","properties":{"config":{"type":"string"}}}}}}}],"conversion":{"strategy":"None"}},"status":{"conditions":null,"acceptedNames":{"plural":"","kind":""},"storedVersions":["v1"]}} 09/01/2025 01:34:59.889 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start init url=https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions resource_version=None 09/01/2025 01:34:59.890 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:34:59.891 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Got ADDED Endpoints kube-system:kube-dns:c3c0e265-ca2a-4647-a9f1-7239a6b01776 09/01/2025 01:34:59.891 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Done init url=https://10.0.0.50:6443/api/v1/endpoints, resource_version=381 09/01/2025 01:34:59.891 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:34:59.891 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:34:59.893 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Got ADDED Namespace None:default:2f58ebf7-edb3-4351-bbe6-c782ef2e1490 09/01/2025 01:34:59.894 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Got ADDED Service kube-system:kube-dns:0aaa8118-7dd7-4cf9-8d00-c56d41939884 09/01/2025 01:34:59.894 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Done init url=https://10.0.0.50:6443/api/v1/services, resource_version=381 09/01/2025 01:34:59.894 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:34:59.894 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:34:59.895 7f490d022900 [contrail-kube-manager] [ERROR]: HTTPError: NetworkMonitor - invalidate resourceVersion None (response 410) 09/01/2025 01:34:59.895 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: HTTPError: NetworkMonitor - invalidate resourceVersion None (response 410) 09/01/2025 01:34:59.896 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:34:59.896 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=False, resource_version=None 09/01/2025 01:34:59.896 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:34:59.896 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:34:59.900 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:34:59.902 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:34:59.902 7f490d022900 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: NetworkMonitor - Retrieved following CRD info = {'kind': 'CustomResourceDefinition', 'apiVersion': 'apiextensions.k8s.io/v1', 'metadata': {'name': 'network-attachment-definitions.k8s.cni.cncf.io', 'uid': '060b80da-99e6-4517-b783-a585521a3460', 'resourceVersion': '382', 'generation': 1, 'creationTimestamp': '2025-09-01T01:34:59Z', 'managedFields': [{'manager': 'python-requests', 'operation': 'Update', 'apiVersion': 'apiextensions.k8s.io/v1', 'time': '2025-09-01T01:34:59Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:spec': {'f:conversion': {'.': {}, 'f:strategy': {}}, 'f:group': {}, 'f:names': {'f:kind': {}, 'f:listKind': {}, 'f:plural': {}, 'f:shortNames': {}, 'f:singular': {}}, 'f:scope': {}, 'f:versions': {}}}}]}, 'spec': {'group': 'k8s.cni.cncf.io', 'names': {'plural': 'network-attachment-definitions', 'singular': 'network-attachment-definition', 'shortNames': ['net-attach-def'], 'kind': 'NetworkAttachmentDefinition', 'listKind': 'NetworkAttachmentDefinitionList'}, 'scope': 'Namespaced', 'versions': [{'name': 'v1', 'served': True, 'storage': True, 'schema': {'openAPIV3Schema': {'type': 'object', 'properties': {'spec': {'type': 'object', 'properties': {'config': {'type': 'string'}}}}}}}], 'conversion': {'strategy': 'None'}}, 'status': {'conditions': None, 'acceptedNames': {'plural': '', 'kind': ''}, 'storedVersions': ['v1']}} 09/01/2025 01:34:59.903 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start init url=https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions resource_version=None 09/01/2025 01:34:59.904 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Got ADDED Namespace None:kube-node-lease:e0db9dca-7fc8-47d9-89a4-88f04377b390 09/01/2025 01:34:59.911 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Got ADDED Namespace None:kube-public:4422e8b3-382f-457d-8515-b4e0ab6cae7b 09/01/2025 01:34:59.915 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Got ADDED Namespace None:kube-system:67c6dbb9-5906-4970-985c-40b5193f4d8a 09/01/2025 01:34:59.915 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Done init url=https://10.0.0.50:6443/api/v1/namespaces, resource_version=381 09/01/2025 01:34:59.915 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:34:59.915 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:34:59.917 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:34:59.949 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Done init url=https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions, resource_version=385 09/01/2025 01:34:59.949 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:34:59.949 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'})(timeout=120) 09/01/2025 01:34:59.951 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['k8s-app'], 'obj_dict': {'display_name': 'k8s-app', 'fq_name': ['k8s-app'], 'id_perms': {'created': '2025-09-01T01:34:59.932934', 'creator': None, 'description': None, 'enable': None, 'last_modified': '2025-09-01T01:34:59.932934', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 13013349798405736565, 'uuid_mslong': 6378882733062310404}}, 'name': 'k8s-app', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x00ff', 'uuid': '5886580c-880d-4a04-b498-b4b1a7e00875'}, 'oper': 'CREATE', 'request-id': 'req-dd1a0a6b-707c-4067-8461-f611ba652626', 'type': 'tag_type', 'uuid': '5886580c-880d-4a04-b498-b4b1a7e00875'} 09/01/2025 01:34:59.953 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'}) 09/01/2025 01:34:59.976 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['k8s-app=kube-dns'], 'obj_dict': {'display_name': 'k8s-app=kube-dns', 'fq_name': ['k8s-app=kube-dns'], 'id_perms': {'created': '2025-09-01T01:34:59.959558', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:34:59.959558', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11471135153113387167, 'uuid_mslong': 7836802123086514403}}, 'name': 'k8s-app=kube-dns', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x00ff0000', 'tag_type_name': 'k8s-app', 'tag_type_refs': [{'to': ['k8s-app'], 'uuid': '5886580c-880d-4a04-b498-b4b1a7e00875'}], 'tag_value': 'kube-dns', 'uuid': '6cc1ea02-8199-48e3-9f31-a8a523d1d09f'}, 'oper': 'CREATE', 'request-id': 'req-dd1a0a6b-707c-4067-8461-f611ba652626', 'type': 'tag', 'uuid': '6cc1ea02-8199-48e3-9f31-a8a523d1d09f'} 09/01/2025 01:35:00.061 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['pod-template-hash'], 'obj_dict': {'display_name': 'pod-template-hash', 'fq_name': ['pod-template-hash'], 'id_perms': {'created': '2025-09-01T01:35:00.054172', 'creator': None, 'description': None, 'enable': None, 'last_modified': '2025-09-01T01:35:00.054172', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 11925374521531939020, 'uuid_mslong': 7275911611303085316}}, 'name': 'pod-template-hash', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0100', 'uuid': '64f93b0f-a0dc-4504-a57f-70f1a967b8cc'}, 'oper': 'CREATE', 'request-id': 'req-ecafa1a4-ff9e-46b3-91a7-b2758a53c848', 'type': 'tag_type', 'uuid': '64f93b0f-a0dc-4504-a57f-70f1a967b8cc'} 09/01/2025 01:35:00.093 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['pod-template-hash=6f867cd986'], 'obj_dict': {'display_name': 'pod-template-hash=6f867cd986', 'fq_name': ['pod-template-hash=6f867cd986'], 'id_perms': {'created': '2025-09-01T01:35:00.081848', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:00.081848', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13788021986930285393, 'uuid_mslong': 6374837122979416772}}, 'name': 'pod-template-hash=6f867cd986', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01000000', 'tag_type_name': 'pod-template-hash', 'tag_type_refs': [{'to': ['pod-template-hash'], 'uuid': '64f93b0f-a0dc-4504-a57f-70f1a967b8cc'}], 'tag_value': '6f867cd986', 'uuid': '5877f896-8829-4ac4-bf58-e4f1ee415b51'}, 'oper': 'CREATE', 'request-id': 'req-ecafa1a4-ff9e-46b3-91a7-b2758a53c848', 'type': 'tag', 'uuid': '5877f896-8829-4ac4-bf58-e4f1ee415b51'} 09/01/2025 01:35:00.193 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['namespace'], 'obj_dict': {'display_name': 'namespace', 'fq_name': ['namespace'], 'id_perms': {'created': '2025-09-01T01:35:00.184338', 'creator': None, 'description': None, 'enable': None, 'last_modified': '2025-09-01T01:35:00.184338', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 12142692889164708494, 'uuid_mslong': 9821876152153425061}}, 'name': 'namespace', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0101', 'uuid': '884e5052-64fc-44a5-a883-82d91352fa8e'}, 'oper': 'CREATE', 'request-id': 'req-c8d73cb7-47c0-47d5-a681-44a5a299ad3c', 'type': 'tag_type', 'uuid': '884e5052-64fc-44a5-a883-82d91352fa8e'} 09/01/2025 01:35:00.221 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['namespace=kube-system'], 'obj_dict': {'display_name': 'namespace=kube-system', 'fq_name': ['namespace=kube-system'], 'id_perms': {'created': '2025-09-01T01:35:00.212283', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:00.212283', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9786979529561711537, 'uuid_mslong': 3566125101318032622}}, 'name': 'namespace=kube-system', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01010000', 'tag_type_name': 'namespace', 'tag_type_refs': [{'to': ['namespace'], 'uuid': '884e5052-64fc-44a5-a883-82d91352fa8e'}], 'tag_value': 'kube-system', 'uuid': '317d6be2-b0aa-4cee-87d2-56074b8ba3b1'}, 'oper': 'CREATE', 'request-id': 'req-c8d73cb7-47c0-47d5-a681-44a5a299ad3c', 'type': 'tag', 'uuid': '317d6be2-b0aa-4cee-87d2-56074b8ba3b1'} 09/01/2025 01:35:00.245 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-2a369105-28c6-49cb-be5b-da7741b80510', 'type': 'global_vrouter_config', 'uuid': '87a687b0-9781-4994-aa87-dbc3f38997c0'} 09/01/2025 01:35:00.430 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-6d90e0f3-59f2-487e-b779-b1873bf72868', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:00.471 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['k8s__kube-system__coredns-6f867cd986-wlkqx'], 'obj_dict': {'display_name': 'k8s__kube-system__coredns-6f867cd986-wlkqx', 'fq_name': ['k8s__kube-system__coredns-6f867cd986-wlkqx'], 'id_perms': {'created': '2025-09-01T01:35:00.459243', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:00.459243', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12979318260264720552, 'uuid_mslong': 11863649037683543068}}, 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'server_type': 'container', 'uuid': 'a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8'}, 'oper': 'CREATE', 'request-id': 'req-c7d91bb6-6d43-4eb5-a9e6-750581fac2d9', 'type': 'virtual_machine', 'uuid': 'a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8'} 09/01/2025 01:35:00.488 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements', 'operand2': {'json_value': '1'}, 'operation': 'size=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-elements-inconsistency', 'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'id_perms': {'created': '2025-09-01T01:35:00.482458', 'creator': None, 'description': 'ContrailConfig has a list of ' 'elements - it should be a dict. ' 'Controllers have inconsistency in ' 'their data.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.482458', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11541890835256865210, 'uuid_mslong': 5907038753012272917}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '51fa0472-f3c3-4f15-a02d-089027d271ba', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-7e298936-cd5c-4a6e-b167-019153a81134', 'type': 'alarm', 'uuid': '51fa0472-f3c3-4f15-a02d-089027d271ba'} 09/01/2025 01:35:00.498 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['k8s__kube-system__coredns-6f867cd986-wlkqx'], 'oper': 'UPDATE', 'request-id': 'req-785a52a2-faa6-404a-a42b-4d1b7282c546', 'type': 'virtual_machine', 'uuid': 'a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8'} 09/01/2025 01:35:00.524 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_ip_address', 'operand2': {'uve_attribute': 'VrouterAgent.control_ip'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-compute', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'id_perms': {'created': '2025-09-01T01:35:00.518275', 'creator': None, 'description': 'Compute Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.518275', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13335252558768919498, 'uuid_mslong': 17592579791041153399}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f425661c-47b9-4977-b910-558c8b8a53ca', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-9ac55cb3-b339-4003-889d-4986b65d2c3c', 'type': 'alarm', 'uuid': 'f425661c-47b9-4977-b910-558c8b8a53ca'} 09/01/2025 01:35:00.600 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d'], 'obj_dict': {'display_name': 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d', 'fq_name': ['default-domain', 'k8s-kube-system', 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d'], 'id_perms': {'created': '2025-09-01T01:35:00.522540', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:00.522540', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10656073166777589261, 'uuid_mslong': 16149988272663368176}}, 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'security_group_refs': [{'to': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg']}], 'uuid': 'e02048c4-86d3-11f0-93e1-fa163e599e0d', 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:e0:20:48:c4:86']}, 'virtual_machine_refs': [{'to': ['k8s__kube-system__coredns-6f867cd986-wlkqx'], 'uuid': 'a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8'}], 'virtual_network_refs': [{'to': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'}]}, 'oper': 'CREATE', 'request-id': 'req-854ab66d-b57e-4e35-8305-1d0f374869a0', 'type': 'virtual_machine_interface', 'uuid': 'e02048c4-86d3-11f0-93e1-fa163e599e0d'} 09/01/2025 01:35:00.618 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.bgp_router_parameters.address', 'operand2': {'uve_attribute': 'BgpRouterState.bgp_router_ip_list'}, 'operation': 'not ' 'in'}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-control', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'id_perms': {'created': '2025-09-01T01:35:00.591651', 'creator': None, 'description': 'Control Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.591651', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13765896304333387515, 'uuid_mslong': 12941592107138171089}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b399c574-35f8-40d1-bf0a-49c102a776fb', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-6f69ecda-97a5-4292-87aa-f9505f715fd1', 'type': 'alarm', 'uuid': 'b399c574-35f8-40d1-bf0a-49c102a776fb'} 09/01/2025 01:35:00.634 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d'], 'oper': 'UPDATE', 'request-id': 'req-854ab66d-b57e-4e35-8305-1d0f374869a0', 'type': 'virtual_machine_interface', 'uuid': 'e02048c4-86d3-11f0-93e1-fa163e599e0d'} 09/01/2025 01:35:00.680 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d'], 'oper': 'UPDATE', 'request-id': 'req-341bddab-ceb6-434c-b328-52a1c8c3f011', 'type': 'virtual_machine_interface', 'uuid': 'e02048c4-86d3-11f0-93e1-fa163e599e0d'} 09/01/2025 01:35:00.697 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_bgp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-bgp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'id_perms': {'created': '2025-09-01T01:35:00.676695', 'creator': None, 'description': 'BGP peer mismatch. Not enough BGP ' 'peers are up.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.676695', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12737114632070279124, 'uuid_mslong': 3323528339732513001}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2e1f8b6a-8ab3-40e9-b0c3-524613faafd4', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-403323e3-fe22-4014-99d0-048dab956108', 'type': 'alarm', 'uuid': '2e1f8b6a-8ab3-40e9-b0c3-524613faafd4'} 09/01/2025 01:35:00.745 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-incorrect', 'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'id_perms': {'created': '2025-09-01T01:35:00.737556', 'creator': None, 'description': 'ContrailConfig missing or ' 'incorrect. Configuration pushed to ' 'Ifmap as ContrailConfig is ' 'missing/incorrect.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.737556', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10437081795927609670, 'uuid_mslong': 1742884343849304489}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '182ff813-97ba-41a9-90d7-f69869f61146', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-ce389d40-228e-4de7-877a-338c3d9f547c', 'type': 'alarm', 'uuid': '182ff813-97ba-41a9-90d7-f69869f61146'} 09/01/2025 01:35:00.752 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncPod: Create IIP from ipam_fq_name [['default-domain', 'k8s-default', 'k8s-pod-ipam']] pod_ipam_subnet_uuid [eada5f2d-448e-4621-9fea-812925eb46e9] vn [k8s-default-pod-network] vmi_fq_name [['default-domain', 'k8s-kube-system', 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d']] 09/01/2025 01:35:00.778 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '[70, ' '90]'}, 'operation': 'range', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-disk-usage-high', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'id_perms': {'created': '2025-09-01T01:35:00.770077', 'creator': None, 'description': 'Disk usage crosses high threshold ' 'limit.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.770077', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10283240903023742618, 'uuid_mslong': 17457322688003264184}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f244de7d-ebcd-4eb8-8eb5-691e4c7f4e9a', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-a4f088e2-148a-4e77-8192-29645683a9a4', 'type': 'alarm', 'uuid': 'f244de7d-ebcd-4eb8-8eb5-691e4c7f4e9a'} 09/01/2025 01:35:00.841 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['coredns-6f867cd986-wlkqx__e03cee52-86d3-11f0-93e1-fa163e599e0d'], 'obj_dict': {'display_name': 'coredns-6f867cd986-wlkqx__e03cee52-86d3-11f0-93e1-fa163e599e0d', 'fq_name': ['coredns-6f867cd986-wlkqx__e03cee52-86d3-11f0-93e1-fa163e599e0d'], 'id_perms': {'created': '2025-09-01T01:35:00.818422', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:00.818422', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10656073166777589261, 'uuid_mslong': 16158051601315205616}}, 'instance_ip_address': '10.47.255.252', 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'subnet_uuid': 'eada5f2d-448e-4621-9fea-812925eb46e9', 'uuid': 'e03cee52-86d3-11f0-93e1-fa163e599e0d', 'virtual_machine_interface_refs': [{'to': ['default-domain', 'k8s-kube-system', 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d'], 'uuid': 'e02048c4-86d3-11f0-93e1-fa163e599e0d'}], 'virtual_network_refs': [{'to': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'}]}, 'oper': 'CREATE', 'request-id': 'req-c80d919e-d72c-4d2d-b5cf-ce84e19a49b9', 'type': 'instance_ip', 'uuid': 'e03cee52-86d3-11f0-93e1-fa163e599e0d'} 09/01/2025 01:35:00.868 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '90'}, 'operation': '>', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-disk-usage-critical', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'id_perms': {'created': '2025-09-01T01:35:00.861644', 'creator': None, 'description': 'Disk usage crosses critical ' 'threshold limit.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.861644', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12788391283575139026, 'uuid_mslong': 5764728618455484405}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '50006e22-b991-47f5-b179-7e1dda32aad2', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-5e38e384-5e93-4efc-b6a4-b22ad6729fbf', 'type': 'alarm', 'uuid': '50006e22-b991-47f5-b179-7e1dda32aad2'} 09/01/2025 01:35:00.879 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncPod - Vrouter 10.0.0.22 (pod_node=an-jenkins-deploy-platform-juju-k8s-1454-1) Not Found for Pod a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8: routers=[] 09/01/2025 01:35:00.886 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['coredns-6f867cd986-wlkqx__e03cee52-86d3-11f0-93e1-fa163e599e0d'], 'oper': 'UPDATE', 'request-id': 'req-cad95a26-b166-494b-93d6-2b3f5d2e7f85', 'type': 'instance_ip', 'uuid': 'e03cee52-86d3-11f0-93e1-fa163e599e0d'} 09/01/2025 01:35:00.904 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-node-status', 'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'id_perms': {'created': '2025-09-01T01:35:00.897537', 'creator': None, 'description': 'Node Failure. NodeStatus UVE not ' 'present.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.897537', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9800546595679803193, 'uuid_mslong': 16768712725561229374}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e8b66f61-23a6-403e-8802-893408289739', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-68e47281-94a2-4cf6-9860-a1a9cffe35b4', 'type': 'alarm', 'uuid': 'e8b66f61-23a6-403e-8802-893408289739'} 09/01/2025 01:35:00.924 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.build_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-partial-sysinfo', 'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'id_perms': {'created': '2025-09-01T01:35:00.919411', 'creator': None, 'description': 'System Info Incomplete.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.919411', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9841848269247336815, 'uuid_mslong': 13773325918658710460}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'bf24aef2-e555-4bbc-8895-44db45c7d16f', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-468f6b69-7fe0-47a7-881d-24e979235447', 'type': 'alarm', 'uuid': 'bf24aef2-e555-4bbc-8895-44db45c7d16f'} 09/01/2025 01:35:00.970 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d'], 'oper': 'UPDATE', 'request-id': 'req-5cb9c65d-a555-4752-be33-f345bd871e71', 'type': 'virtual_machine_interface', 'uuid': 'e02048c4-86d3-11f0-93e1-fa163e599e0d'} 09/01/2025 01:35:01.002 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_status', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_status.state', 'operand2': {'json_value': '"Functional"'}, 'operation': '!=', 'variables': ['NodeStatus.process_status.module_id', 'NodeStatus.process_status.instance_id']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'id_perms': {'created': '2025-09-01T01:35:00.993525', 'creator': None, 'description': 'Process(es) reporting as ' 'non-functional.', 'enable': True, 'last_modified': '2025-09-01T01:35:00.993525', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13655082713157644973, 'uuid_mslong': 7422783952441593023}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '670306ad-59c0-4cbf-bd80-99614056b2ad', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-553602d1-d201-4b11-8569-96d2a7801aee', 'type': 'alarm', 'uuid': '670306ad-59c0-4cbf-bd80-99614056b2ad'} 09/01/2025 01:35:01.007 7f490d0792c0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:35:01.031 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_info.process_state', 'operand2': {'json_value': '"PROCESS_STATE_RUNNING"'}, 'operation': '!=', 'variables': ['NodeStatus.process_info.process_name']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-status', 'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'id_perms': {'created': '2025-09-01T01:35:01.022450', 'creator': None, 'description': 'Process Failure.', 'enable': True, 'last_modified': '2025-09-01T01:35:01.022450', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10351739637149028015, 'uuid_mslong': 894098479270019581}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0c6879d4-6b1b-41fd-8fa8-c45aa7994eaf', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-d87fd2d0-ee20-4f9c-aa86-fa97c52766aa', 'type': 'alarm', 'uuid': '0c6879d4-6b1b-41fd-8fa8-c45aa7994eaf'} 09/01/2025 01:35:01.058 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=14 timeout=120) 09/01/2025 01:35:01.059 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:35:01.059 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:35:01.059 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncPod - Vrouter 10.0.0.22 (pod_node=an-jenkins-deploy-platform-juju-k8s-1454-1) Not Found for Pod a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8: routers=[] 09/01/2025 01:35:01.059 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:35:01.059 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=13 timeout=120) 09/01/2025 01:35:01.059 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=contrail event_type=ADDED kind=Namespace ns=None) 09/01/2025 01:35:01.059 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got ADDED Namespace contrail:a28020d1-63f5-47e0-90e8-7a75e7378bf9 09/01/2025 01:35:01.061 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received MODIFIED Pod kube-system:coredns-6f867cd986-wlkqx:a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8 09/01/2025 01:35:01.061 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Got MODIFIED Pod kube-system:coredns-6f867cd986-wlkqx:a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8 09/01/2025 01:35:01.061 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:35:01.062 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:35:01.062 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:35:01.066 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.connected_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'id_perms': {'created': '2025-09-01T01:35:01.060992', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tor agent does not exist we look ' 'for non-empty value for ' 'connected_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:35:01.060992', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10335860996523724715, 'uuid_mslong': 16042134643040011702}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dea11c76-8563-45b6-8f70-5ad0899c8fab', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-a70d16ac-9e75-4a7b-abc2-b8ad36bbe09e', 'type': 'alarm', 'uuid': 'dea11c76-8563-45b6-8f70-5ad0899c8fab'} 09/01/2025 01:35:01.090 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kubernetes.io__slash__metadata.name'], 'obj_dict': {'display_name': 'kubernetes.io__slash__metadata.name', 'fq_name': ['kubernetes.io__slash__metadata.name'], 'id_perms': {'created': '2025-09-01T01:35:01.087385', 'creator': None, 'description': None, 'enable': None, 'last_modified': '2025-09-01T01:35:01.087385', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 13828626881080767734, 'uuid_mslong': 14178771421114287520}}, 'name': 'kubernetes.io__slash__metadata.name', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0102', 'uuid': 'c4c51d7c-32fa-45a0-bfe9-26e1923cb4f6'}, 'oper': 'CREATE', 'request-id': 'req-e4e030dc-7426-4998-a3af-f25f1827bff9', 'type': 'tag_type', 'uuid': 'c4c51d7c-32fa-45a0-bfe9-26e1923cb4f6'} 09/01/2025 01:35:01.118 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kubernetes.io__slash__metadata.name=contrail'], 'obj_dict': {'display_name': 'kubernetes.io__slash__metadata.name=contrail', 'fq_name': ['kubernetes.io__slash__metadata.name=contrail'], 'id_perms': {'created': '2025-09-01T01:35:01.111225', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:01.111225', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12675199517905652019, 'uuid_mslong': 11957672635128762339}}, 'name': 'kubernetes.io__slash__metadata.name=contrail', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01020000', 'tag_type_name': 'kubernetes.io__slash__metadata.name', 'tag_type_refs': [{'to': ['kubernetes.io__slash__metadata.name'], 'uuid': 'c4c51d7c-32fa-45a0-bfe9-26e1923cb4f6'}], 'tag_value': 'contrail', 'uuid': 'a5f22fe8-3d42-4fe3-afe7-5ace31f80933'}, 'oper': 'CREATE', 'request-id': 'req-e4e030dc-7426-4998-a3af-f25f1827bff9', 'type': 'tag', 'uuid': 'a5f22fe8-3d42-4fe3-afe7-5ace31f80933'} 09/01/2025 01:35:01.185 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '!='}]}, {'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-tsn-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'id_perms': {'created': '2025-09-01T01:35:01.130566', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tsn agent does not exist we look ' 'for non-empty value for ' 'tsn_agent_list', 'enable': True, 'last_modified': '2025-09-01T01:35:01.130566', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10504271381337467867, 'uuid_mslong': 3142675033853412581}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2b9d064a-ed52-44e5-91c6-ab2ad4ebebdb', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-253fdcfe-1152-4fe9-8a2b-7295bef192e3', 'type': 'alarm', 'uuid': '2b9d064a-ed52-44e5-91c6-ab2ad4ebebdb'} 09/01/2025 01:35:01.228 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'StorageCluster.info_stats.status', 'operand2': {'json_value': '0'}, 'operation': '!=', 'variables': ['StorageCluster.info_stats.health_summary']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-storage-cluster-state', 'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'id_perms': {'created': '2025-09-01T01:35:01.217585', 'creator': None, 'description': 'Storage Cluster warning/errors.', 'enable': True, 'last_modified': '2025-09-01T01:35:01.217585', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12284578432857240966, 'uuid_mslong': 2852633633655180775}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2796972d-0859-4de7-aa7b-96ff89e17586', 'uve_keys': {'uve_key': ['storage-cluster']}}, 'oper': 'CREATE', 'request-id': 'req-aa6e16b3-5471-478a-9bfe-77fcf366581b', 'type': 'alarm', 'uuid': '2796972d-0859-4de7-aa7b-96ff89e17586'} 09/01/2025 01:35:01.391 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.down_interface_count', 'operand2': {'json_value': '1'}, 'operation': '>=', 'variables': ['VrouterAgent.error_intf_list', 'VrouterAgent.no_config_intf_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-interface', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'id_perms': {'created': '2025-09-01T01:35:01.386361', 'creator': None, 'description': 'Vrouter interface(s) down.', 'enable': True, 'last_modified': '2025-09-01T01:35:01.386361', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12208880729113631627, 'uuid_mslong': 17929068975300495646}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f8d0d93d-a243-491e-a96e-a85673ee1b8b', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-9fc5f925-daa3-46cb-b392-b79e0fde1670', 'type': 'alarm', 'uuid': 'f8d0d93d-a243-491e-a96e-a85673ee1b8b'} 09/01/2025 01:35:01.600 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['namespace=contrail'], 'obj_dict': {'display_name': 'namespace=contrail', 'fq_name': ['namespace=contrail'], 'id_perms': {'created': '2025-09-01T01:35:01.557786', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:01.557786', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9297889043599944578, 'uuid_mslong': 764438863256043651}}, 'name': 'namespace=contrail', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01010001', 'tag_type_name': 'namespace', 'tag_type_refs': [{'to': ['namespace'], 'uuid': '884e5052-64fc-44a5-a883-82d91352fa8e'}], 'tag_value': 'contrail', 'uuid': '0a9bd518-90e1-4083-8108-bcd2a4c56b82'}, 'oper': 'CREATE', 'request-id': 'req-386a6bfc-4958-462d-9f83-0968d86a25f4', 'type': 'tag', 'uuid': '0a9bd518-90e1-4083-8108-bcd2a4c56b82'} 09/01/2025 01:35:01.663 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_xmpp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'id_perms': {'created': '2025-09-01T01:35:01.645266', 'creator': None, 'description': 'XMPP peer mismatch.', 'enable': True, 'last_modified': '2025-09-01T01:35:01.645266', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12068159765698629765, 'uuid_mslong': 7356697359278424505}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '66183d45-53d8-41b9-a77a-b75df720d085', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-92fc4d7a-14e9-4658-a1ae-6a28b3678823', 'type': 'alarm', 'uuid': '66183d45-53d8-41b9-a77a-b75df720d085'} 09/01/2025 01:35:01.705 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'XmppPeerInfoData.close_reason', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'XmppPeerInfoData.state_info.state', 'operand2': {'json_value': '"Established"'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-close-reason', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'id_perms': {'created': '2025-09-01T01:35:01.691703', 'creator': None, 'description': 'XMPP connection closed towards ' 'peer, ' 'alarm has close reason', 'enable': True, 'last_modified': '2025-09-01T01:35:01.691703', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12165997184363612657, 'uuid_mslong': 4383944909124291530}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3cd6e6aa-ef3c-4bca-a8d6-4dfb206549f1', 'uve_keys': {'uve_key': ['xmpp-peer']}}, 'oper': 'CREATE', 'request-id': 'req-b024a142-709f-492d-9da3-c5bfe05e542d', 'type': 'alarm', 'uuid': '3cd6e6aa-ef3c-4bca-a8d6-4dfb206549f1'} 09/01/2025 01:35:01.729 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail'], 'obj_dict': {'display_name': 'k8s-contrail', 'fq_name': ['default-domain', 'k8s-contrail'], 'id_perms': {'created': '2025-09-01T01:35:01.717377', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:01.717377', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10022379383818498906, 'uuid_mslong': 581816361761063834}}, 'parent_type': 'domain', 'parent_uuid': 'ce056a4b-12ea-41a5-896d-393d506f2011', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '081306e5-ff47-439a-8b16-a4f2e0b3335a'}, 'oper': 'CREATE', 'request-id': 'req-f6923dc9-5523-457c-b447-e80fcbd284bc', 'type': 'project', 'uuid': '081306e5-ff47-439a-8b16-a4f2e0b3335a'} 09/01/2025 01:35:01.769 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-core-files', 'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'id_perms': {'created': '2025-09-01T01:35:01.757037', 'creator': None, 'description': 'A core file has been generated on ' 'the node.', 'enable': True, 'last_modified': '2025-09-01T01:35:01.757037', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11501329277269896852, 'uuid_mslong': 7179776699111329448}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '63a3b0de-2528-4ea8-9f9c-ee0a8078ee94', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-a623d8e3-f755-46c3-a9fa-86f1648b62b1', 'type': 'alarm', 'uuid': '63a3b0de-2528-4ea8-9f9c-ee0a8078ee94'} 09/01/2025 01:35:01.797 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'display_name': 'default-application-policy-set', 'fq_name': ['default-domain', 'k8s-contrail', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:35:01.791455', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:01.791455', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11424969983852188087, 'uuid_mslong': 3187153430437121841}}, 'parent_type': 'project', 'parent_uuid': '081306e5-ff47-439a-8b16-a4f2e0b3335a', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '2c3b0b28-a1df-4f31-9e8d-a5ab110291b7'}, 'oper': 'CREATE', 'request-id': 'req-f6923dc9-5523-457c-b447-e80fcbd284bc', 'type': 'application_policy_set', 'uuid': '2c3b0b28-a1df-4f31-9e8d-a5ab110291b7'} 09/01/2025 01:35:01.840 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail'], 'oper': 'UPDATE', 'request-id': 'req-f6923dc9-5523-457c-b447-e80fcbd284bc', 'type': 'project', 'uuid': '081306e5-ff47-439a-8b16-a4f2e0b3335a'} 09/01/2025 01:35:01.885 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks', 'operand2': {'json_value': '300'}, 'operation': '>='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-pending-cassandra-compaction-tasks', 'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'id_perms': {'created': '2025-09-01T01:35:01.847333', 'creator': None, 'description': 'Pending compaction tasks in ' 'cassandra crossed the configured ' 'threshold.', 'enable': True, 'last_modified': '2025-09-01T01:35:01.847333', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12785828124223818586, 'uuid_mslong': 1599492486595233073}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '163289f1-2c6a-4531-b170-62efcf8f4b5a', 'uve_keys': {'uve_key': ['database-nodeconfig-database-node']}}, 'oper': 'CREATE', 'request-id': 'req-45d7d2da-c7ae-4f84-9f98-ca6666f35674', 'type': 'alarm', 'uuid': '163289f1-2c6a-4531-b170-62efcf8f4b5a'} 09/01/2025 01:35:01.929 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.running_package_version', 'operand2': {'uve_attribute': 'NodeStatus.installed_package_version'}, 'operation': '!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-package-version-mismatch', 'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'id_perms': {'created': '2025-09-01T01:35:01.919901', 'creator': None, 'description': 'There is a mismatch between ' 'installed and running package ' 'version.', 'enable': True, 'last_modified': '2025-09-01T01:35:01.919901', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9569444448425120023, 'uuid_mslong': 17242802562263173967}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ef4abd92-f7e4-4b4f-84cd-7f0656369117', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-3cdce8d0-2fa6-434f-9793-383dff488d4f', 'type': 'alarm', 'uuid': 'ef4abd92-f7e4-4b4f-84cd-7f0656369117'} 09/01/2025 01:35:01.944 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail', 'default'], 'obj_dict': {'display_name': 'default', 'fq_name': ['default-domain', 'k8s-contrail', 'default'], 'id_perms': {'created': '2025-09-01T01:35:01.920792', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:35:01.920792', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12418792943944759266, 'uuid_mslong': 12529932026945814864}}, 'parent_type': 'project', 'parent_uuid': '081306e5-ff47-439a-8b16-a4f2e0b3335a', 'perms2': {'global_access': 0, 'owner': '081306e5-ff47-439a-8b16-a4f2e0b3335a', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': [{'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'd804800b-1a50-43a7-bbf4-00825bf2493d', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-contrail:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'bbd99afb-2454-4294-8e57-8389a1db15a8', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-contrail:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'faa114d9-5d7d-4d3e-887b-950566a31e78', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '8c6e98be-4225-4ef3-bec9-0b5e2d468aa6', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}]}, 'security_group_id': 8000006, 'uuid': 'ade342ca-c5e5-4150-ac58-6a628af477e2'}, 'oper': 'CREATE', 'request-id': 'req-e8beff0e-d485-41cb-9653-01f21116aaf1', 'type': 'security_group', 'uuid': 'ade342ca-c5e5-4150-ac58-6a628af477e2'} 09/01/2025 01:35:01.993 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:35:01.982979', 'creator': None, 'description': 'Agent resource usage exceeded ' 'configured watermark for resource.', 'enable': True, 'last_modified': '2025-09-01T01:35:01.982979', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11703449765111169089, 'uuid_mslong': 12023298412754059803}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a6db5634-48b9-421b-a26b-018dc8148041', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-bda61db9-3971-4fce-ada9-03bd9e2b9951', 'type': 'alarm', 'uuid': 'a6db5634-48b9-421b-a26b-018dc8148041'} 09/01/2025 01:35:02.017 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail', 'default', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000006', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'd804800b-1a50-43a7-bbf4-00825bf2493d'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000006', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'bbd99afb-2454-4294-8e57-8389a1db15a8'}], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-contrail', 'default', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:02.004892', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:02.004892', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9548772609716880718, 'uuid_mslong': 14927954596232578009}}, 'parent_type': 'security-group', 'parent_uuid': 'ade342ca-c5e5-4150-ac58-6a628af477e2', 'perms2': {'global_access': 0, 'owner': '081306e5-ff47-439a-8b16-a4f2e0b3335a', 'owner_access': 7, 'share': []}, 'uuid': 'cf2abf9d-139a-43d9-8484-0e18d327854e'}, 'oper': 'CREATE', 'request-id': 'req-3bfe52a8-b579-41fa-9985-6b1ed0de9582', 'type': 'access_control_list', 'uuid': 'cf2abf9d-139a-43d9-8484-0e18d327854e'} 09/01/2025 01:35:02.024 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail'], 'oper': 'UPDATE', 'request-id': 'req-5414dd4a-0dbb-4323-bd25-63dcd92fad2d', 'type': 'project', 'uuid': '081306e5-ff47-439a-8b16-a4f2e0b3335a'} 09/01/2025 01:35:02.087 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_table_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-table-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'id_perms': {'created': '2025-09-01T01:35:02.077045', 'creator': None, 'description': 'Agent resource usage exceeded table ' 'size for resource in vrouter.', 'enable': True, 'last_modified': '2025-09-01T01:35:02.077045', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13572308632227373667, 'uuid_mslong': 6614552712968424813}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5bcb9ca5-a494-4d6d-bc5a-86cab5e16263', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-5350116d-a370-45ea-a441-44398ce65320', 'type': 'alarm', 'uuid': '5bcb9ca5-a494-4d6d-bc5a-86cab5e16263'} 09/01/2025 01:35:02.169 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail', 'default', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'faa114d9-5d7d-4d3e-887b-950566a31e78'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '8c6e98be-4225-4ef3-bec9-0b5e2d468aa6'}], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-contrail', 'default', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:02.094615', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:02.094615', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12448090761444631064, 'uuid_mslong': 3778302967327313713}}, 'parent_type': 'security-group', 'parent_uuid': 'ade342ca-c5e5-4150-ac58-6a628af477e2', 'perms2': {'global_access': 0, 'owner': '081306e5-ff47-439a-8b16-a4f2e0b3335a', 'owner_access': 7, 'share': []}, 'uuid': '346f3a87-cd59-4f31-acc0-80983fbaa218'}, 'oper': 'CREATE', 'request-id': 'req-9ee39db1-9e38-4b0d-846b-682b1482786b', 'type': 'access_control_list', 'uuid': '346f3a87-cd59-4f31-acc0-80983fbaa218'} 09/01/2025 01:35:02.251 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.l3mh_phy_interface_down', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_l3mh_intf_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-l3mh-physical-interface-link-down', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'id_perms': {'created': '2025-09-01T01:35:02.218693', 'creator': None, 'description': 'On l3 multihomed compute one or ' 'more physical interfaces are ' 'Inactive.', 'enable': True, 'last_modified': '2025-09-01T01:35:02.218693', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10910313226223040477, 'uuid_mslong': 10914525326012924032}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '97782ef4-d501-4080-9769-3812ed4537dd', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-41017a40-8c14-4dbc-adcf-aed02fca1acb', 'type': 'alarm', 'uuid': '97782ef4-d501-4080-9769-3812ed4537dd'} 09/01/2025 01:35:02.260 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-8c7c3952-7c75-4929-816b-276916e02baa', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:35:02.354 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-8c7c3952-7c75-4929-816b-276916e02baa', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:35:02.375 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-e158efc6-5d82-4115-87ed-1b5df6f86915', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:35:02.403 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-e158efc6-5d82-4115-87ed-1b5df6f86915', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:35:02.494 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail', 'k8s-contrail-default-sg'], 'obj_dict': {'display_name': 'k8s-contrail-default-sg', 'fq_name': ['default-domain', 'k8s-contrail', 'k8s-contrail-default-sg'], 'id_perms': {'created': '2025-09-01T01:35:02.440898', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:35:02.440898', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10716835512065823629, 'uuid_mslong': 11418210244986487825}}, 'parent_type': 'project', 'parent_uuid': '081306e5-ff47-439a-8b16-a4f2e0b3335a', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': [{'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '21fdef34-3bf5-42fa-96b1-22b0c5228755', 'src_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '31e479d7-1eaf-4aba-9e13-86ad6cd34c10', 'src_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '232937ba-3215-42cc-bea1-15bb5eeff8dd', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'cf245d60-d348-471f-b7e4-a265c7e2fc87', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}]}, 'security_group_id': 8000007, 'uuid': '9e75a1b8-b725-4411-94b9-d91e34f9938d'}, 'oper': 'CREATE', 'request-id': 'req-17a260cb-66df-4abe-aa94-0a775f7ca667', 'type': 'security_group', 'uuid': '9e75a1b8-b725-4411-94b9-d91e34f9938d'} 09/01/2025 01:35:02.551 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-f1e0bc91-b510-4c29-8f03-74b1320fe568', 'type': 'global_vrouter_config', 'uuid': '87a687b0-9781-4994-aa87-dbc3f38997c0'} 09/01/2025 01:35:02.563 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_security_group_id': None}, 'fq_name': ['default-domain', 'k8s-contrail', 'k8s-contrail-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-e7a802d7-8225-44cc-92df-e864f964ac58', 'type': 'security_group', 'uuid': '9e75a1b8-b725-4411-94b9-d91e34f9938d'} 09/01/2025 01:35:02.591 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail', 'k8s-contrail-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-3b559cce-98b5-4e6c-90a0-f1224781a986', 'type': 'security_group', 'uuid': '9e75a1b8-b725-4411-94b9-d91e34f9938d'} 09/01/2025 01:35:02.600 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail', 'k8s-contrail-default-sg', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '21fdef34-3bf5-42fa-96b1-22b0c5228755'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '31e479d7-1eaf-4aba-9e13-86ad6cd34c10'}], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-contrail', 'k8s-contrail-default-sg', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:02.576967', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:02.576967', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13403080425643194522, 'uuid_mslong': 5016664225494878536}}, 'parent_type': 'security-group', 'parent_uuid': '9e75a1b8-b725-4411-94b9-d91e34f9938d', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '459ec588-9e08-4948-ba01-4ea27274389a'}, 'oper': 'CREATE', 'request-id': 'req-efffa46b-2a53-4867-a5f2-a2c1d4c53658', 'type': 'access_control_list', 'uuid': '459ec588-9e08-4948-ba01-4ea27274389a'} 09/01/2025 01:35:02.725 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-contrail', 'k8s-contrail-default-sg', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '232937ba-3215-42cc-bea1-15bb5eeff8dd'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'cf245d60-d348-471f-b7e4-a265c7e2fc87'}], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-contrail', 'k8s-contrail-default-sg', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:02.652534', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:02.652534', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12395505328244393664, 'uuid_mslong': 16623783802522192305}}, 'parent_type': 'security-group', 'parent_uuid': '9e75a1b8-b725-4411-94b9-d91e34f9938d', 'perms2': {'global_access': 0, 'owner': '081306e5-ff47-439a-8b16-a4f2e0b3335a', 'owner_access': 7, 'share': []}, 'uuid': 'e6b38b4a-b43a-49b1-ac05-ae6bfd7806c0'}, 'oper': 'CREATE', 'request-id': 'req-86814ae2-c3de-44e8-a9c3-8d60164d274e', 'type': 'access_control_list', 'uuid': 'e6b38b4a-b43a-49b1-ac05-ae6bfd7806c0'} 09/01/2025 01:35:02.805 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-contrail-ingress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-contrail-ingress', 'endpoint_1': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842753], 'tags': ['namespace=contrail'], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-contrail-ingress'], 'id_perms': {'created': '2025-09-01T01:35:02.748868', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:02.748868', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10183644137596290291, 'uuid_mslong': 5575263468403573073}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=contrail'], 'uuid': '0a9bd518-90e1-4083-8108-bcd2a4c56b82'}], 'uuid': '4d5f5095-e368-4151-8d53-92634c6a84f3'}, 'oper': 'CREATE', 'request-id': 'req-7562bb48-db8d-4a88-81bf-f41930c935bd', 'type': 'firewall_rule', 'uuid': '4d5f5095-e368-4151-8d53-92634c6a84f3'} 09/01/2025 01:35:02.877 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:02.880 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:02.880 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:02.880 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:02.880 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:02.880 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:02.881 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:02.881 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:02.881 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:02.881 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:02.881 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:02.881 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:02.881 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:02.881 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:02.931 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-2b864465-9bda-450c-9ea5-037acee3ce31', 'type': 'global_vrouter_config', 'uuid': '87a687b0-9781-4994-aa87-dbc3f38997c0'} 09/01/2025 01:35:02.956 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-36286e64-75c0-4420-bcc6-90f5138fbab0', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:03.194 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-contrail-egress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-contrail-egress', 'endpoint_1': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842753], 'tags': ['namespace=contrail'], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-contrail-egress'], 'id_perms': {'created': '2025-09-01T01:35:03.125206', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:03.125206', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11604150138806812443, 'uuid_mslong': 12778824617337704182}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=contrail'], 'uuid': '0a9bd518-90e1-4083-8108-bcd2a4c56b82'}], 'uuid': 'b1578148-cb6e-4ef6-a10a-3911dfe13f1b'}, 'oper': 'CREATE', 'request-id': 'req-32c4d195-9bb9-4cef-b82d-7a4b8a39c7af', 'type': 'firewall_rule', 'uuid': 'b1578148-cb6e-4ef6-a10a-3911dfe13f1b'} 09/01/2025 01:35:03.252 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-57cf9ce7-8537-4ed5-a884-fe86d3d00ede', 'type': 'global_system_config', 'uuid': '70eaa5ad-608f-40c9-a193-2be391842efd'} 09/01/2025 01:35:03.321 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-e5632fa9-9725-4744-86b1-f4682e824cfc', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:03.359 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=13 timeout=120) 09/01/2025 01:35:03.359 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=kubernetes event_type=ADDED kind=Endpoints ns=default) 09/01/2025 01:35:03.359 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncEndpoints - Got ADDED Endpoints default:kubernetes:578b072c-0e11-408a-a97f-e0b0b4c830b3 09/01/2025 01:35:03.370 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Add/Modify endpoints event received while service kubernetes does not exist 09/01/2025 01:35:03.370 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=12 timeout=120) 09/01/2025 01:35:03.370 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=kubernetes event_type=ADDED kind=Service ns=default) 09/01/2025 01:35:03.370 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncService - Got ADDED Service default:kubernetes:4149e3a7-961c-4071-90b8-b63acefd96b1 09/01/2025 01:35:03.395 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['k8s-svc'], 'obj_dict': {'display_name': 'k8s-svc', 'fq_name': ['k8s-svc'], 'id_perms': {'created': '2025-09-01T01:35:03.390359', 'creator': None, 'description': None, 'enable': None, 'last_modified': '2025-09-01T01:35:03.390359', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 11543464585065071630, 'uuid_mslong': 15268846470947556803}}, 'name': 'k8s-svc', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0103', 'uuid': 'd3e5d6f7-aece-4dc3-a032-9fe1561b040e'}, 'oper': 'CREATE', 'request-id': 'req-43818550-ab00-4cb8-8c28-dd5a32a8bb02', 'type': 'tag_type', 'uuid': 'd3e5d6f7-aece-4dc3-a032-9fe1561b040e'} 09/01/2025 01:35:03.433 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['k8s-svc=kubernetes'], 'obj_dict': {'display_name': 'k8s-svc=kubernetes', 'fq_name': ['k8s-svc=kubernetes'], 'id_perms': {'created': '2025-09-01T01:35:03.423866', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:03.423866', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9232761476190750890, 'uuid_mslong': 7821842526510074911}}, 'name': 'k8s-svc=kubernetes', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01030000', 'tag_type_name': 'k8s-svc', 'tag_type_refs': [{'to': ['k8s-svc'], 'uuid': 'd3e5d6f7-aece-4dc3-a032-9fe1561b040e'}], 'tag_value': 'kubernetes', 'uuid': '6c8cc456-01ad-4c1f-8021-5ba53442c8aa'}, 'oper': 'CREATE', 'request-id': 'req-43818550-ab00-4cb8-8c28-dd5a32a8bb02', 'type': 'tag', 'uuid': '6c8cc456-01ad-4c1f-8021-5ba53442c8aa'} 09/01/2025 01:35:03.525 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default'], 'oper': 'UPDATE', 'request-id': 'req-e519e413-4c52-4bd9-bb51-6e15fb5ebf86', 'type': 'project', 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'} 09/01/2025 01:35:03.676 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Create LB Interface ['default-domain', 'k8s-default', 'k8s__Service__kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'] 09/01/2025 01:35:03.769 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s__Service__kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'obj_dict': {'display_name': 'default__kubernetes', 'fq_name': ['default-domain', 'k8s-default', 'k8s__Service__kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'id_perms': {'created': '2025-09-01T01:35:03.714936', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:03.714936', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10490939060604764201, 'uuid_mslong': 697245358317782078}}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'security_group_refs': [{'to': ['default-domain', 'k8s-default', 'k8s-default-default-sg']}], 'uuid': '09ad1cf5-8fd2-443e-9197-4d7e039e0029', 'virtual_machine_interface_device_owner': 'K8S:LOADBALANCER', 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:09:ad:1c:f5:8f']}, 'virtual_network_refs': [{'to': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'}]}, 'oper': 'CREATE', 'request-id': 'req-ebf64703-2423-41fb-8784-1254e6b22ab9', 'type': 'virtual_machine_interface', 'uuid': '09ad1cf5-8fd2-443e-9197-4d7e039e0029'} 09/01/2025 01:35:03.794 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s__Service__kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'oper': 'UPDATE', 'request-id': 'req-ebf64703-2423-41fb-8784-1254e6b22ab9', 'type': 'virtual_machine_interface', 'uuid': '09ad1cf5-8fd2-443e-9197-4d7e039e0029'} 09/01/2025 01:35:03.847 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Create LB VMI InstanceIp ['kubernetes__7a7d590e-3340-4e88-a363-525aacd0cc23'] 09/01/2025 01:35:03.903 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kubernetes__7a7d590e-3340-4e88-a363-525aacd0cc23'], 'obj_dict': {'display_name': 'kubernetes', 'fq_name': ['kubernetes__7a7d590e-3340-4e88-a363-525aacd0cc23'], 'id_perms': {'created': '2025-09-01T01:35:03.879882', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:03.879882', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11773344400276507683, 'uuid_mslong': 8826308762240306824}}, 'instance_ip_address': '10.96.0.1', 'perms2': {'global_access': 0, 'owner': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'owner_access': 7, 'share': []}, 'subnet_uuid': 'cdf5a7bb-3fee-45f9-9299-6a9e2ef0e400', 'uuid': '7a7d590e-3340-4e88-a363-525aacd0cc23', 'virtual_machine_interface_refs': [{'to': ['default-domain', 'k8s-default', 'k8s__Service__kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'uuid': '09ad1cf5-8fd2-443e-9197-4d7e039e0029'}], 'virtual_network_refs': [{'to': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'}]}, 'oper': 'CREATE', 'request-id': 'req-1d81df02-2873-48de-a3e5-27ed63c314a4', 'type': 'instance_ip', 'uuid': '7a7d590e-3340-4e88-a363-525aacd0cc23'} 09/01/2025 01:35:03.914 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Created LB VMI InstanceIp ['kubernetes__7a7d590e-3340-4e88-a363-525aacd0cc23'] with VIP 10.96.0.1 09/01/2025 01:35:04.004 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'obj_dict': {'display_name': 'default__kubernetes', 'fq_name': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'id_perms': {'created': '2025-09-01T01:35:03.939097', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:03.939097', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10428285300873467569, 'uuid_mslong': 4704541594682671217}}, 'loadbalancer_properties': {'admin_state': True, 'operating_status': 'ONLINE', 'provisioning_status': 'ACTIVE', 'status': None, 'vip_address': '10.96.0.1', 'vip_subnet_id': None}, 'loadbalancer_provider': 'native', 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'service_appliance_set_refs': [{'to': ['default-global-system-config', 'native'], 'uuid': 'ce9684c7-029d-4e16-ad7f-6b14554d8775'}], 'uuid': '4149e3a7-961c-4071-90b8-b63acefd96b1', 'virtual_machine_interface_refs': [{'to': ['default-domain', 'k8s-default', 'k8s__Service__kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'uuid': '09ad1cf5-8fd2-443e-9197-4d7e039e0029'}]}, 'oper': 'CREATE', 'request-id': 'req-0be6ff65-919f-4513-9db1-ad436c8d62fb', 'type': 'loadbalancer', 'uuid': '4149e3a7-961c-4071-90b8-b63acefd96b1'} 09/01/2025 01:35:04.066 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'oper': 'UPDATE', 'request-id': 'req-3ba112e4-6a8f-4de0-b8d7-a0443d433b47', 'type': 'loadbalancer', 'uuid': '4149e3a7-961c-4071-90b8-b63acefd96b1'} 09/01/2025 01:35:04.088 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default'], 'oper': 'UPDATE', 'request-id': 'req-c4362afb-39c1-4736-8e15-c1eb7f6b633f', 'type': 'project', 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'} 09/01/2025 01:35:04.144 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default'], 'oper': 'UPDATE', 'request-id': 'req-c077d094-eca3-4f1c-aa1d-8f898a4dd81c', 'type': 'project', 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'} 09/01/2025 01:35:04.208 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kubernetes__7a7d590e-3340-4e88-a363-525aacd0cc23', '6e324ad2-40d0-4016-9315-3daed4aa8660'], 'obj_dict': {'display_name': '6e324ad2-40d0-4016-9315-3daed4aa8660', 'floating_ip_address': '10.96.0.1', 'floating_ip_traffic_direction': 'ingress', 'fq_name': ['kubernetes__7a7d590e-3340-4e88-a363-525aacd0cc23', '6e324ad2-40d0-4016-9315-3daed4aa8660'], 'id_perms': {'created': '2025-09-01T01:35:04.197352', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:04.197352', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10598445119187879520, 'uuid_mslong': 7940491359898583062}}, 'parent_type': 'instance-ip', 'parent_uuid': '7a7d590e-3340-4e88-a363-525aacd0cc23', 'perms2': {'global_access': 0, 'owner': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'owner_access': 7, 'share': []}, 'project_refs': [{'to': ['default-domain', 'k8s-default'], 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'}], 'uuid': '6e324ad2-40d0-4016-9315-3daed4aa8660'}, 'oper': 'CREATE', 'request-id': 'req-6ed18d80-3769-462c-a82c-86b3504280a7', 'type': 'floating_ip', 'uuid': '6e324ad2-40d0-4016-9315-3daed4aa8660'} 09/01/2025 01:35:04.219 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1-TCP-443-11d02361-e609-4206-93f6-a60460b35529'], 'obj_dict': {'display_name': 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1-TCP-443-11d02361-e609-4206-93f6-a60460b35529', 'fq_name': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1-TCP-443-11d02361-e609-4206-93f6-a60460b35529'], 'id_perms': {'created': '2025-09-01T01:35:04.198842', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:04.198842', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10661891705578673449, 'uuid_mslong': 1283564797178757638}}, 'loadbalancer_listener_properties': {'admin_state': True, 'connection_limit': None, 'default_tls_container': None, 'protocol': 'TCP', 'protocol_port': 443, 'sni_containers': []}, 'loadbalancer_refs': [{'to': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1'], 'uuid': '4149e3a7-961c-4071-90b8-b63acefd96b1'}], 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '11d02361-e609-4206-93f6-a60460b35529'}, 'oper': 'CREATE', 'request-id': 'req-39ff537b-90f1-4945-9ff4-d0ea178df0cc', 'type': 'loadbalancer_listener', 'uuid': '11d02361-e609-4206-93f6-a60460b35529'} 09/01/2025 01:35:04.225 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1-TCP-443-11d02361-e609-4206-93f6-a60460b35529'], 'oper': 'UPDATE', 'request-id': 'req-d62472a7-40d7-4304-961e-f17fc8ce3fb8', 'type': 'loadbalancer_listener', 'uuid': '11d02361-e609-4206-93f6-a60460b35529'} 09/01/2025 01:35:04.287 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default'], 'oper': 'UPDATE', 'request-id': 'req-667400c6-66ca-45e3-a5f8-7645de4c80d1', 'type': 'project', 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'} 09/01/2025 01:35:04.388 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1-TCP-443-11d02361-e609-4206-93f6-a60460b35529'], 'obj_dict': {'display_name': 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1-TCP-443-11d02361-e609-4206-93f6-a60460b35529', 'fq_name': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1-TCP-443-11d02361-e609-4206-93f6-a60460b35529'], 'id_perms': {'created': '2025-09-01T01:35:04.378765', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:04.378765', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11940756889999979924, 'uuid_mslong': 47868323585213139}}, 'loadbalancer_listener_refs': [{'to': ['default-domain', 'k8s-default', 'kubernetes__4149e3a7-961c-4071-90b8-b63acefd96b1-TCP-443-11d02361-e609-4206-93f6-a60460b35529'], 'uuid': '11d02361-e609-4206-93f6-a60460b35529'}], 'loadbalancer_pool_properties': {'admin_state': True, 'loadbalancer_method': None, 'persistence_cookie_name': None, 'protocol': 'TCP', 'session_persistence': None, 'status': None, 'status_description': None, 'subnet_id': None}, 'parent_type': 'project', 'parent_uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '00aa0ffc-974a-42d3-a5b6-172069bf3594'}, 'oper': 'CREATE', 'request-id': 'req-ae91cf19-b028-4cc5-a49d-8a7f13fc7c9e', 'type': 'loadbalancer_pool', 'uuid': '00aa0ffc-974a-42d3-a5b6-172069bf3594'} 09/01/2025 01:35:04.465 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-b9fd73f9-712b-4ccf-b991-429312f9ebdc', 'type': 'global_vrouter_config', 'uuid': '87a687b0-9781-4994-aa87-dbc3f38997c0'} 09/01/2025 01:35:04.466 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=11 timeout=120) 09/01/2025 01:35:04.466 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=kube-dns event_type=ADDED kind=Endpoints ns=kube-system) 09/01/2025 01:35:04.466 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncEndpoints - Got ADDED Endpoints kube-system:kube-dns:c3c0e265-ca2a-4647-a9f1-7239a6b01776 09/01/2025 01:35:04.485 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Add/Modify endpoints event received while service kube-dns does not exist 09/01/2025 01:35:04.485 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=10 timeout=120) 09/01/2025 01:35:04.485 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:35:04.485 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:35:04.485 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=9 timeout=120) 09/01/2025 01:35:04.485 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=default event_type=ADDED kind=Namespace ns=None) 09/01/2025 01:35:04.486 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got ADDED Namespace default:2f58ebf7-edb3-4351-bbe6-c782ef2e1490 09/01/2025 01:35:04.514 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kubernetes.io__slash__metadata.name=default'], 'obj_dict': {'display_name': 'kubernetes.io__slash__metadata.name=default', 'fq_name': ['kubernetes.io__slash__metadata.name=default'], 'id_perms': {'created': '2025-09-01T01:35:04.498091', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:04.498091', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13662651753165036361, 'uuid_mslong': 1818953846815737330}}, 'name': 'kubernetes.io__slash__metadata.name=default', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01020001', 'tag_type_name': 'kubernetes.io__slash__metadata.name', 'tag_type_refs': [{'to': ['kubernetes.io__slash__metadata.name'], 'uuid': 'c4c51d7c-32fa-45a0-bfe9-26e1923cb4f6'}], 'tag_value': 'default', 'uuid': '193e38e2-ee48-45f2-bd9b-7d61b5451b49'}, 'oper': 'CREATE', 'request-id': 'req-1eb9ae72-3722-444e-bbec-f19cadc5c2a6', 'type': 'tag', 'uuid': '193e38e2-ee48-45f2-bd9b-7d61b5451b49'} 09/01/2025 01:35:04.603 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['namespace=default'], 'obj_dict': {'display_name': 'namespace=default', 'fq_name': ['namespace=default'], 'id_perms': {'created': '2025-09-01T01:35:04.589424', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:04.589424', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10845566140282948963, 'uuid_mslong': 18086324725812776608}}, 'name': 'namespace=default', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01010002', 'tag_type_name': 'namespace', 'tag_type_refs': [{'to': ['namespace'], 'uuid': '884e5052-64fc-44a5-a883-82d91352fa8e'}], 'tag_value': 'default', 'uuid': 'faff8883-3e6f-4ea0-9683-30f13b91cd63'}, 'oper': 'CREATE', 'request-id': 'req-423e7e90-c62d-4292-a7c5-f57bb4a9127d', 'type': 'tag', 'uuid': 'faff8883-3e6f-4ea0-9683-30f13b91cd63'} 09/01/2025 01:35:04.734 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default'], 'oper': 'UPDATE', 'request-id': 'req-7e6f9af7-71b2-4c01-bfe5-c70b2589413b', 'type': 'project', 'uuid': '9023d5fc-34ad-4e7e-9d9e-872a06007bf8'} 09/01/2025 01:35:04.859 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-96f09b0e-4843-46a4-bcc3-14d7074bee54', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:35:04.904 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-96f09b0e-4843-46a4-bcc3-14d7074bee54', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:35:04.935 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-11dc5953-d3ad-42b5-ba91-597bc14c0006', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:35:04.963 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-11dc5953-d3ad-42b5-ba91-597bc14c0006', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:35:05.098 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-8b5248da-cf34-4c23-99f8-c358e451cdee', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:35:05.117 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-8b5248da-cf34-4c23-99f8-c358e451cdee', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:35:05.162 7f490d0ae680 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:35:05.221 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-532a29a8-81ca-4ca6-afe9-0d0a1bd27472', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:35:05.251 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-532a29a8-81ca-4ca6-afe9-0d0a1bd27472', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:35:05.424 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-service-np'], 'oper': 'UPDATE', 'request-id': 'req-f4014d9f-0e90-4d56-878d-92b1d34140c3', 'type': 'network_policy', 'uuid': '1f1a62da-5984-46dc-82ee-e07966ea5188'} 09/01/2025 01:35:05.439 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-a969ab3e-d4dd-4022-bdbc-9a1f72472875', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:35:05.458 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-a969ab3e-d4dd-4022-bdbc-9a1f72472875', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:35:05.779 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_security_group_id': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-ea757534-9276-4848-bf9e-208ea86cff25', 'type': 'security_group', 'uuid': '4de733ab-781c-4219-bcab-efd8adb97fdf'} 09/01/2025 01:35:05.795 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-10ea4a51-8f7f-43da-bee8-bb494de9a14b', 'type': 'access_control_list', 'uuid': '0a0f9403-fbe0-4cf4-8dd8-61b6703c9a68'} 09/01/2025 01:35:05.818 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-71caae77-7989-4f26-b852-4d8f72616c2e', 'type': 'access_control_list', 'uuid': 'e9c77d30-5ca4-4541-9585-66a4ccc4f9e1'} 09/01/2025 01:35:05.830 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_security_group_id': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-99a26f15-1cc0-4bef-9a6e-db5b3355427e', 'type': 'security_group', 'uuid': '4de733ab-781c-4219-bcab-efd8adb97fdf'} 09/01/2025 01:35:06.025 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-default-ingress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-default-ingress', 'endpoint_1': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842754], 'tags': ['namespace=default'], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-default-ingress'], 'id_perms': {'created': '2025-09-01T01:35:05.975858', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:05.975858', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13323639037654214571, 'uuid_mslong': 10651579802644989837}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=default'], 'uuid': 'faff8883-3e6f-4ea0-9683-30f13b91cd63'}], 'uuid': '93d20365-6161-478d-b8e7-131cf2e827ab'}, 'oper': 'CREATE', 'request-id': 'req-2fc07c57-fbcd-40c9-a93e-fd2c7c97d1fb', 'type': 'firewall_rule', 'uuid': '93d20365-6161-478d-b8e7-131cf2e827ab'} 09/01/2025 01:35:06.113 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-2a936ac1-be50-4966-854f-efc81508be55', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:06.319 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-default-egress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-default-egress', 'endpoint_1': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842754], 'tags': ['namespace=default'], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-default-egress'], 'id_perms': {'created': '2025-09-01T01:35:06.261460', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:06.261460', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12047502222840008587, 'uuid_mslong': 17769216293693834069}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=default'], 'uuid': 'faff8883-3e6f-4ea0-9683-30f13b91cd63'}], 'uuid': 'f698f012-e2b4-4355-a731-5370f71b8f8b'}, 'oper': 'CREATE', 'request-id': 'req-31f11afa-0ead-4690-9c99-719bee03805c', 'type': 'firewall_rule', 'uuid': 'f698f012-e2b4-4355-a731-5370f71b8f8b'} 09/01/2025 01:35:06.452 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=8 timeout=120) 09/01/2025 01:35:06.452 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=kube-dns event_type=ADDED kind=Service ns=kube-system) 09/01/2025 01:35:06.452 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncService - Got ADDED Service kube-system:kube-dns:0aaa8118-7dd7-4cf9-8d00-c56d41939884 09/01/2025 01:35:06.453 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-f7649ee7-e03c-4af4-94d7-7b3cecade25a', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:06.491 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['k8s-svc=kube-dns'], 'obj_dict': {'display_name': 'k8s-svc=kube-dns', 'fq_name': ['k8s-svc=kube-dns'], 'id_perms': {'created': '2025-09-01T01:35:06.479406', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:06.479406', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11013834266846936573, 'uuid_mslong': 13924478071625763768}}, 'name': 'k8s-svc=kube-dns', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01030001', 'tag_type_name': 'k8s-svc', 'tag_type_refs': [{'to': ['k8s-svc'], 'uuid': 'd3e5d6f7-aece-4dc3-a032-9fe1561b040e'}], 'tag_value': 'kube-dns', 'uuid': 'c13daf07-f5f8-4bb8-98d8-ffe9726fbdfd'}, 'oper': 'CREATE', 'request-id': 'req-615b3138-4664-4587-a225-9d3958bd9152', 'type': 'tag', 'uuid': 'c13daf07-f5f8-4bb8-98d8-ffe9726fbdfd'} 09/01/2025 01:35:06.596 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-bd0c6ce6-1fc9-4d54-b986-551ec0ac81d7', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:06.717 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Create LB Interface ['default-domain', 'k8s-kube-system', 'k8s__Service__kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'] 09/01/2025 01:35:06.821 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'k8s__Service__kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'obj_dict': {'display_name': 'kube-system__kube-dns', 'fq_name': ['default-domain', 'k8s-kube-system', 'k8s__Service__kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'id_perms': {'created': '2025-09-01T01:35:06.764506', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:06.764506', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10524097296904489957, 'uuid_mslong': 4096956869932238070}}, 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'security_group_refs': [{'to': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg']}], 'uuid': '38db5091-0c68-44f6-920d-1abb83a40be5', 'virtual_machine_interface_device_owner': 'K8S:LOADBALANCER', 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:38:db:50:91:0c']}, 'virtual_network_refs': [{'to': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'}]}, 'oper': 'CREATE', 'request-id': 'req-5af837c2-d2d1-4da8-9099-29707c80c392', 'type': 'virtual_machine_interface', 'uuid': '38db5091-0c68-44f6-920d-1abb83a40be5'} 09/01/2025 01:35:06.856 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'k8s__Service__kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'oper': 'UPDATE', 'request-id': 'req-5af837c2-d2d1-4da8-9099-29707c80c392', 'type': 'virtual_machine_interface', 'uuid': '38db5091-0c68-44f6-920d-1abb83a40be5'} 09/01/2025 01:35:06.880 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:06.881 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:06.882 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:06.882 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:06.915 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Create LB VMI InstanceIp ['kube-dns__62fa6a29-784f-494c-91c4-4ca8a0b13fc2'] 09/01/2025 01:35:06.950 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kube-dns__62fa6a29-784f-494c-91c4-4ca8a0b13fc2'], 'obj_dict': {'display_name': 'kube-dns', 'fq_name': ['kube-dns__62fa6a29-784f-494c-91c4-4ca8a0b13fc2'], 'id_perms': {'created': '2025-09-01T01:35:06.936398', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:06.936398', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10503604518069026754, 'uuid_mslong': 7132129686239267148}}, 'instance_ip_address': '10.105.226.84', 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'subnet_uuid': 'cdf5a7bb-3fee-45f9-9299-6a9e2ef0e400', 'uuid': '62fa6a29-784f-494c-91c4-4ca8a0b13fc2', 'virtual_machine_interface_refs': [{'to': ['default-domain', 'k8s-kube-system', 'k8s__Service__kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'uuid': '38db5091-0c68-44f6-920d-1abb83a40be5'}], 'virtual_network_refs': [{'to': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'}]}, 'oper': 'CREATE', 'request-id': 'req-f5541a6a-8df6-4267-8dfe-656f78dc6123', 'type': 'instance_ip', 'uuid': '62fa6a29-784f-494c-91c4-4ca8a0b13fc2'} 09/01/2025 01:35:06.966 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Created LB VMI InstanceIp ['kube-dns__62fa6a29-784f-494c-91c4-4ca8a0b13fc2'] with VIP 10.105.226.84 09/01/2025 01:35:07.036 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'obj_dict': {'display_name': 'kube-system__kube-dns', 'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'id_perms': {'created': '2025-09-01T01:35:06.982232', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:06.982232', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10160337832390137988, 'uuid_mslong': 768568628610551033}}, 'loadbalancer_properties': {'admin_state': True, 'operating_status': 'ONLINE', 'provisioning_status': 'ACTIVE', 'status': None, 'vip_address': '10.105.226.84', 'vip_subnet_id': None}, 'loadbalancer_provider': 'native', 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'service_appliance_set_refs': [{'to': ['default-global-system-config', 'native'], 'uuid': 'ce9684c7-029d-4e16-ad7f-6b14554d8775'}], 'uuid': '0aaa8118-7dd7-4cf9-8d00-c56d41939884', 'virtual_machine_interface_refs': [{'to': ['default-domain', 'k8s-kube-system', 'k8s__Service__kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'uuid': '38db5091-0c68-44f6-920d-1abb83a40be5'}]}, 'oper': 'CREATE', 'request-id': 'req-e3172508-51f9-47f1-8974-4188a938672d', 'type': 'loadbalancer', 'uuid': '0aaa8118-7dd7-4cf9-8d00-c56d41939884'} 09/01/2025 01:35:07.064 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-6ed18d80-3769-462c-a82c-86b3504280a7', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:07.100 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'oper': 'UPDATE', 'request-id': 'req-11cd59a6-02d3-4968-a81c-bbf3f6b2bba3', 'type': 'loadbalancer', 'uuid': '0aaa8118-7dd7-4cf9-8d00-c56d41939884'} 09/01/2025 01:35:07.119 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-e5683e35-5c34-4c9a-8220-141ab3216dd0', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:07.145 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kube-dns__62fa6a29-784f-494c-91c4-4ca8a0b13fc2', 'e2e22085-46d7-4e9f-9530-11e688ef29d7'], 'obj_dict': {'display_name': 'e2e22085-46d7-4e9f-9530-11e688ef29d7', 'floating_ip_address': '10.105.226.84', 'floating_ip_traffic_direction': 'ingress', 'fq_name': ['kube-dns__62fa6a29-784f-494c-91c4-4ca8a0b13fc2', 'e2e22085-46d7-4e9f-9530-11e688ef29d7'], 'id_perms': {'created': '2025-09-01T01:35:07.137763', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:07.137763', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10750111992370899415, 'uuid_mslong': 16348665354099576479}}, 'parent_type': 'instance-ip', 'parent_uuid': '62fa6a29-784f-494c-91c4-4ca8a0b13fc2', 'perms2': {'global_access': 0, 'owner': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'owner_access': 7, 'share': []}, 'project_refs': [{'to': ['default-domain', 'k8s-kube-system'], 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'}], 'uuid': 'e2e22085-46d7-4e9f-9530-11e688ef29d7'}, 'oper': 'CREATE', 'request-id': 'req-73079181-47b0-4dec-98fa-025964e5bba1', 'type': 'floating_ip', 'uuid': 'e2e22085-46d7-4e9f-9530-11e688ef29d7'} 09/01/2025 01:35:07.194 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-UDP-53-a8fccf8f-1536-430b-9b8a-f74dd27122df'], 'obj_dict': {'display_name': 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-UDP-53-a8fccf8f-1536-430b-9b8a-f74dd27122df', 'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-UDP-53-a8fccf8f-1536-430b-9b8a-f74dd27122df'], 'id_perms': {'created': '2025-09-01T01:35:07.186572', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:07.186572', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11208042536280072927, 'uuid_mslong': 12176835705946129163}}, 'loadbalancer_listener_properties': {'admin_state': True, 'connection_limit': None, 'default_tls_container': None, 'protocol': 'UDP', 'protocol_port': 53, 'sni_containers': []}, 'loadbalancer_refs': [{'to': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'uuid': '0aaa8118-7dd7-4cf9-8d00-c56d41939884'}], 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'a8fccf8f-1536-430b-9b8a-f74dd27122df'}, 'oper': 'CREATE', 'request-id': 'req-f8ccf411-b268-4e0d-ad61-4749ce522a22', 'type': 'loadbalancer_listener', 'uuid': 'a8fccf8f-1536-430b-9b8a-f74dd27122df'} 09/01/2025 01:35:07.210 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-UDP-53-a8fccf8f-1536-430b-9b8a-f74dd27122df'], 'oper': 'UPDATE', 'request-id': 'req-e040d7a9-7962-4c7e-89e1-24d1f5ba7333', 'type': 'loadbalancer_listener', 'uuid': 'a8fccf8f-1536-430b-9b8a-f74dd27122df'} 09/01/2025 01:35:07.272 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-0fe3f21a-58d0-42f3-967c-fd6c676916a3', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:07.367 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-UDP-53-a8fccf8f-1536-430b-9b8a-f74dd27122df'], 'obj_dict': {'display_name': 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-UDP-53-a8fccf8f-1536-430b-9b8a-f74dd27122df', 'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-UDP-53-a8fccf8f-1536-430b-9b8a-f74dd27122df'], 'id_perms': {'created': '2025-09-01T01:35:07.357069', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:07.357069', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12369833172442127356, 'uuid_mslong': 328866569655238908}}, 'loadbalancer_listener_refs': [{'to': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-UDP-53-a8fccf8f-1536-430b-9b8a-f74dd27122df'], 'uuid': 'a8fccf8f-1536-430b-9b8a-f74dd27122df'}], 'loadbalancer_pool_properties': {'admin_state': True, 'loadbalancer_method': None, 'persistence_cookie_name': None, 'protocol': 'UDP', 'session_persistence': None, 'status': None, 'status_description': None, 'subnet_id': None}, 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '04905e67-16cd-40fc-abaa-79bb6e419bfc'}, 'oper': 'CREATE', 'request-id': 'req-c991a278-f364-4a90-ac3a-364e8750245e', 'type': 'loadbalancer_pool', 'uuid': '04905e67-16cd-40fc-abaa-79bb6e419bfc'} 09/01/2025 01:35:07.392 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-c7c7a965-24fc-40d4-995d-1c21b4ccb98f', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:07.472 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-53-096a7ee3-a48c-4713-949f-f75b8435fbb3'], 'obj_dict': {'display_name': 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-53-096a7ee3-a48c-4713-949f-f75b8435fbb3', 'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-53-096a7ee3-a48c-4713-949f-f75b8435fbb3'], 'id_perms': {'created': '2025-09-01T01:35:07.463282', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:07.463282', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10709550411342543795, 'uuid_mslong': 678494210056013587}}, 'loadbalancer_listener_properties': {'admin_state': True, 'connection_limit': None, 'default_tls_container': None, 'protocol': 'TCP', 'protocol_port': 53, 'sni_containers': []}, 'loadbalancer_refs': [{'to': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'uuid': '0aaa8118-7dd7-4cf9-8d00-c56d41939884'}], 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '096a7ee3-a48c-4713-949f-f75b8435fbb3'}, 'oper': 'CREATE', 'request-id': 'req-689cc893-4473-47ec-b007-f31fa198879b', 'type': 'loadbalancer_listener', 'uuid': '096a7ee3-a48c-4713-949f-f75b8435fbb3'} 09/01/2025 01:35:07.487 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-53-096a7ee3-a48c-4713-949f-f75b8435fbb3'], 'oper': 'UPDATE', 'request-id': 'req-672a2903-2517-4396-bb13-c6d3b693205c', 'type': 'loadbalancer_listener', 'uuid': '096a7ee3-a48c-4713-949f-f75b8435fbb3'} 09/01/2025 01:35:07.552 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-6acf1880-92cd-41ec-82c2-dc00cee45367', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:07.597 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-53-096a7ee3-a48c-4713-949f-f75b8435fbb3'], 'obj_dict': {'display_name': 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-53-096a7ee3-a48c-4713-949f-f75b8435fbb3', 'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-53-096a7ee3-a48c-4713-949f-f75b8435fbb3'], 'id_perms': {'created': '2025-09-01T01:35:07.591632', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:07.591632', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12997013705438630066, 'uuid_mslong': 5282566568977188615}}, 'loadbalancer_listener_refs': [{'to': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-53-096a7ee3-a48c-4713-949f-f75b8435fbb3'], 'uuid': '096a7ee3-a48c-4713-949f-f75b8435fbb3'}], 'loadbalancer_pool_properties': {'admin_state': True, 'loadbalancer_method': None, 'persistence_cookie_name': None, 'protocol': 'TCP', 'session_persistence': None, 'status': None, 'status_description': None, 'subnet_id': None}, 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '494f724e-663d-4707-b45e-ab1a9d8d90b2'}, 'oper': 'CREATE', 'request-id': 'req-665d527a-7f15-4ca7-8d71-91ad818ac7a0', 'type': 'loadbalancer_pool', 'uuid': '494f724e-663d-4707-b45e-ab1a9d8d90b2'} 09/01/2025 01:35:07.666 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-0d33716b-4d2c-4b04-80a7-9505fc4c84b1', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:07.697 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-9153-d29b75ec-b179-4d0b-9e50-01234a890d63'], 'obj_dict': {'display_name': 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-9153-d29b75ec-b179-4d0b-9e50-01234a890d63', 'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-9153-d29b75ec-b179-4d0b-9e50-01234a890d63'], 'id_perms': {'created': '2025-09-01T01:35:07.689058', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:07.689058', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11407619107215445347, 'uuid_mslong': 15175853028805266699}}, 'loadbalancer_listener_properties': {'admin_state': True, 'connection_limit': None, 'default_tls_container': None, 'protocol': 'TCP', 'protocol_port': 9153, 'sni_containers': []}, 'loadbalancer_refs': [{'to': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884'], 'uuid': '0aaa8118-7dd7-4cf9-8d00-c56d41939884'}], 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'd29b75ec-b179-4d0b-9e50-01234a890d63'}, 'oper': 'CREATE', 'request-id': 'req-8b9baf47-a7af-4cb8-beb3-49a05b8164db', 'type': 'loadbalancer_listener', 'uuid': 'd29b75ec-b179-4d0b-9e50-01234a890d63'} 09/01/2025 01:35:07.745 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-9153-d29b75ec-b179-4d0b-9e50-01234a890d63'], 'oper': 'UPDATE', 'request-id': 'req-9f31a74c-b813-4e82-ba2d-4d9c64acf448', 'type': 'loadbalancer_listener', 'uuid': 'd29b75ec-b179-4d0b-9e50-01234a890d63'} 09/01/2025 01:35:07.771 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-e2309acc-fa5d-4797-adc5-34a2c98e1b29', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:07.865 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-9153-d29b75ec-b179-4d0b-9e50-01234a890d63'], 'obj_dict': {'display_name': 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-9153-d29b75ec-b179-4d0b-9e50-01234a890d63', 'fq_name': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-9153-d29b75ec-b179-4d0b-9e50-01234a890d63'], 'id_perms': {'created': '2025-09-01T01:35:07.857798', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:07.857798', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12107355557576793757, 'uuid_mslong': 12602419387071219849}}, 'loadbalancer_listener_refs': [{'to': ['default-domain', 'k8s-kube-system', 'kube-dns__0aaa8118-7dd7-4cf9-8d00-c56d41939884-TCP-9153-d29b75ec-b179-4d0b-9e50-01234a890d63'], 'uuid': 'd29b75ec-b179-4d0b-9e50-01234a890d63'}], 'loadbalancer_pool_properties': {'admin_state': True, 'loadbalancer_method': None, 'persistence_cookie_name': None, 'protocol': 'TCP', 'session_persistence': None, 'status': None, 'status_description': None, 'subnet_id': None}, 'parent_type': 'project', 'parent_uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': 'aee4c9a9-7ce0-4889-a805-f7bb6ab75e9d'}, 'oper': 'CREATE', 'request-id': 'req-43656287-d4ff-46d5-a655-da77ddad12ea', 'type': 'loadbalancer_pool', 'uuid': 'aee4c9a9-7ce0-4889-a805-f7bb6ab75e9d'} 09/01/2025 01:35:07.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=7 timeout=120) 09/01/2025 01:35:07.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:35:07.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:35:07.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:35:07.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=6 timeout=120) 09/01/2025 01:35:07.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=kube-node-lease event_type=ADDED kind=Namespace ns=None) 09/01/2025 01:35:07.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got ADDED Namespace kube-node-lease:e0db9dca-7fc8-47d9-89a4-88f04377b390 09/01/2025 01:35:07.928 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kubernetes.io__slash__metadata.name=kube-node-lease'], 'obj_dict': {'display_name': 'kubernetes.io__slash__metadata.name=kube-node-lease', 'fq_name': ['kubernetes.io__slash__metadata.name=kube-node-lease'], 'id_perms': {'created': '2025-09-01T01:35:07.921452', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:07.921452', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13374079485892415267, 'uuid_mslong': 15718588142150699747}}, 'name': 'kubernetes.io__slash__metadata.name=kube-node-lease', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01020002', 'tag_type_name': 'kubernetes.io__slash__metadata.name', 'tag_type_refs': [{'to': ['kubernetes.io__slash__metadata.name'], 'uuid': 'c4c51d7c-32fa-45a0-bfe9-26e1923cb4f6'}], 'tag_value': 'kube-node-lease', 'uuid': 'da23a4a2-778f-4ae3-b99a-466efa74cf23'}, 'oper': 'CREATE', 'request-id': 'req-96744b7d-1c4d-455b-a918-7b19300b1a83', 'type': 'tag', 'uuid': 'da23a4a2-778f-4ae3-b99a-466efa74cf23'} 09/01/2025 01:35:08.006 7f490d10c720 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:35:08.028 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['namespace=kube-node-lease'], 'obj_dict': {'display_name': 'namespace=kube-node-lease', 'fq_name': ['namespace=kube-node-lease'], 'id_perms': {'created': '2025-09-01T01:35:08.016756', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:08.016756', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13078349777989594095, 'uuid_mslong': 11288639297156038718}}, 'name': 'namespace=kube-node-lease', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01010003', 'tag_type_name': 'namespace', 'tag_type_refs': [{'to': ['namespace'], 'uuid': '884e5052-64fc-44a5-a883-82d91352fa8e'}], 'tag_value': 'kube-node-lease', 'uuid': '9ca94da1-8880-403e-b57f-a1d4bd60ebef'}, 'oper': 'CREATE', 'request-id': 'req-56619a5e-043b-439e-a45a-1ae7eb3863d1', 'type': 'tag', 'uuid': '9ca94da1-8880-403e-b57f-a1d4bd60ebef'} 09/01/2025 01:35:08.123 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease'], 'obj_dict': {'display_name': 'k8s-kube-node-lease', 'fq_name': ['default-domain', 'k8s-kube-node-lease'], 'id_perms': {'created': '2025-09-01T01:35:08.117860', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:08.117860', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13243892127741515929, 'uuid_mslong': 10235971928760011485}}, 'parent_type': 'domain', 'parent_uuid': 'ce056a4b-12ea-41a5-896d-393d506f2011', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '8e0d7a3c-989c-4add-b7cb-c1b895a01899'}, 'oper': 'CREATE', 'request-id': 'req-67755f30-ee01-481f-82ec-6bdf3ec8ed6d', 'type': 'project', 'uuid': '8e0d7a3c-989c-4add-b7cb-c1b895a01899'} 09/01/2025 01:35:08.145 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'display_name': 'default-application-policy-set', 'fq_name': ['default-domain', 'k8s-kube-node-lease', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:35:08.141506', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:08.141506', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11599686833657261955, 'uuid_mslong': 6557066945879952406}}, 'parent_type': 'project', 'parent_uuid': '8e0d7a3c-989c-4add-b7cb-c1b895a01899', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '5aff61a5-7da1-4c16-a0fa-5db7a9eaaf83'}, 'oper': 'CREATE', 'request-id': 'req-67755f30-ee01-481f-82ec-6bdf3ec8ed6d', 'type': 'application_policy_set', 'uuid': '5aff61a5-7da1-4c16-a0fa-5db7a9eaaf83'} 09/01/2025 01:35:08.162 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease'], 'oper': 'UPDATE', 'request-id': 'req-67755f30-ee01-481f-82ec-6bdf3ec8ed6d', 'type': 'project', 'uuid': '8e0d7a3c-989c-4add-b7cb-c1b895a01899'} 09/01/2025 01:35:08.241 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease', 'default'], 'obj_dict': {'display_name': 'default', 'fq_name': ['default-domain', 'k8s-kube-node-lease', 'default'], 'id_perms': {'created': '2025-09-01T01:35:08.189203', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:35:08.189203', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10355378182601732884, 'uuid_mslong': 3631824606228794343}}, 'parent_type': 'project', 'parent_uuid': '8e0d7a3c-989c-4add-b7cb-c1b895a01899', 'perms2': {'global_access': 0, 'owner': '8e0d7a3c-989c-4add-b7cb-c1b895a01899', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': [{'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '1bf977a9-083f-4f52-adc5-794fce9a63e5', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-kube-node-lease:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '2393b834-6985-4b82-8b4e-39ac9c7dc3ce', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-kube-node-lease:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '1c72fa75-47db-4cb2-adee-4ab2a119da26', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'ae9fe320-160f-4589-a736-93a29502bcb9', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}]}, 'security_group_id': 8000008, 'uuid': '3266d53c-b4ca-4be7-8fb5-b19788ce7314'}, 'oper': 'CREATE', 'request-id': 'req-adcf92cc-3838-4d8c-b02f-1e4b7123e458', 'type': 'security_group', 'uuid': '3266d53c-b4ca-4be7-8fb5-b19788ce7314'} 09/01/2025 01:35:08.260 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease'], 'oper': 'UPDATE', 'request-id': 'req-c8dd4699-30f4-4302-8658-d1c9c7e961c2', 'type': 'project', 'uuid': '8e0d7a3c-989c-4add-b7cb-c1b895a01899'} 09/01/2025 01:35:08.506 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease', 'default', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000008', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '1bf977a9-083f-4f52-adc5-794fce9a63e5'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000008', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '2393b834-6985-4b82-8b4e-39ac9c7dc3ce'}], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-node-lease', 'default', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:08.443304', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:08.443304', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13779271877774155511, 'uuid_mslong': 1901039727771534284}}, 'parent_type': 'security-group', 'parent_uuid': '3266d53c-b4ca-4be7-8fb5-b19788ce7314', 'perms2': {'global_access': 0, 'owner': '8e0d7a3c-989c-4add-b7cb-c1b895a01899', 'owner_access': 7, 'share': []}, 'uuid': '1a61d98f-6dba-43cc-bf39-cec462421ef7'}, 'oper': 'CREATE', 'request-id': 'req-627a7f7f-388f-416a-bef6-8db21c7f4970', 'type': 'access_control_list', 'uuid': '1a61d98f-6dba-43cc-bf39-cec462421ef7'} 09/01/2025 01:35:08.512 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-3bda41c6-5d8b-415a-9fd7-14913639ec04', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:35:08.562 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-3bda41c6-5d8b-415a-9fd7-14913639ec04', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:35:08.612 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease', 'default', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '1c72fa75-47db-4cb2-adee-4ab2a119da26'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'ae9fe320-160f-4589-a736-93a29502bcb9'}], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-node-lease', 'default', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:08.552827', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:08.552827', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12223284962926760408, 'uuid_mslong': 9767097960267923502}}, 'parent_type': 'security-group', 'parent_uuid': '3266d53c-b4ca-4be7-8fb5-b19788ce7314', 'perms2': {'global_access': 0, 'owner': '8e0d7a3c-989c-4add-b7cb-c1b895a01899', 'owner_access': 7, 'share': []}, 'uuid': '878bb3d8-b836-402e-a9a1-d4e97b9f91d8'}, 'oper': 'CREATE', 'request-id': 'req-390b984d-1730-424d-bdf5-3edc2e41b892', 'type': 'access_control_list', 'uuid': '878bb3d8-b836-402e-a9a1-d4e97b9f91d8'} 09/01/2025 01:35:08.626 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-de605c4f-b2e0-41c2-9951-be9556a27795', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:35:08.644 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-de605c4f-b2e0-41c2-9951-be9556a27795', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:35:08.730 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease', 'k8s-kube-node-lease-default-sg'], 'obj_dict': {'display_name': 'k8s-kube-node-lease-default-sg', 'fq_name': ['default-domain', 'k8s-kube-node-lease', 'k8s-kube-node-lease-default-sg'], 'id_perms': {'created': '2025-09-01T01:35:08.671049', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:35:08.671049', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12277904507868437021, 'uuid_mslong': 17077533445816337081}}, 'parent_type': 'project', 'parent_uuid': '8e0d7a3c-989c-4add-b7cb-c1b895a01899', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': [{'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '01335bfd-0746-431c-918c-c51187711b8b', 'src_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '7dae5209-fb44-4db7-9366-8c4f5e9a37b1', 'src_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '27af981d-28c1-44b2-97f0-b4212fb494b7', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '824ed245-0ba2-44e1-9f82-f7c6fe54ba48', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}]}, 'security_group_id': 8000009, 'uuid': 'ecff9630-35bc-42b9-aa63-e11949aa5a1d'}, 'oper': 'CREATE', 'request-id': 'req-ac7d6189-024e-4169-b43d-76c3a6d20aef', 'type': 'security_group', 'uuid': 'ecff9630-35bc-42b9-aa63-e11949aa5a1d'} 09/01/2025 01:35:08.813 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease', 'k8s-kube-node-lease-default-sg', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '01335bfd-0746-431c-918c-c51187711b8b'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '7dae5209-fb44-4db7-9366-8c4f5e9a37b1'}], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-node-lease', 'k8s-kube-node-lease-default-sg', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:08.756795', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:08.756795', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11466293722866954989, 'uuid_mslong': 6670521070751402776}}, 'parent_type': 'security-group', 'parent_uuid': 'ecff9630-35bc-42b9-aa63-e11949aa5a1d', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '5c927392-68e5-4b18-9f20-7563d1c6b2ed'}, 'oper': 'CREATE', 'request-id': 'req-f69b0936-4451-4dea-ad2e-7fc5f9534daf', 'type': 'access_control_list', 'uuid': '5c927392-68e5-4b18-9f20-7563d1c6b2ed'} 09/01/2025 01:35:08.844 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_security_group_id': None}, 'fq_name': ['default-domain', 'k8s-kube-node-lease', 'k8s-kube-node-lease-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-877457c7-d08c-4c98-99ac-53ff0411afdc', 'type': 'security_group', 'uuid': 'ecff9630-35bc-42b9-aa63-e11949aa5a1d'} 09/01/2025 01:35:08.883 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease', 'k8s-kube-node-lease-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-26beae75-376d-42fc-a43f-c3482dc143d1', 'type': 'security_group', 'uuid': 'ecff9630-35bc-42b9-aa63-e11949aa5a1d'} 09/01/2025 01:35:08.951 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-node-lease', 'k8s-kube-node-lease-default-sg', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '27af981d-28c1-44b2-97f0-b4212fb494b7'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '824ed245-0ba2-44e1-9f82-f7c6fe54ba48'}], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-node-lease', 'k8s-kube-node-lease-default-sg', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:08.901065', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:08.901065', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12329592412285012411, 'uuid_mslong': 51089021332179352}}, 'parent_type': 'security-group', 'parent_uuid': 'ecff9630-35bc-42b9-aa63-e11949aa5a1d', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '00b58131-b86d-4998-ab1b-82f9579989bb'}, 'oper': 'CREATE', 'request-id': 'req-3d943d03-6797-4d8a-bd47-f83455eca619', 'type': 'access_control_list', 'uuid': '00b58131-b86d-4998-ab1b-82f9579989bb'} 09/01/2025 01:35:09.125 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-kube-node-lease-ingress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-kube-node-lease-ingress', 'endpoint_1': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842755], 'tags': ['namespace=kube-node-lease'], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-kube-node-lease-ingress'], 'id_perms': {'created': '2025-09-01T01:35:09.068780', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:09.068780', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9791100812219705833, 'uuid_mslong': 17402610047194647621}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=kube-node-lease'], 'uuid': '9ca94da1-8880-403e-b57f-a1d4bd60ebef'}], 'uuid': 'f1827da2-8bae-4c45-87e0-fa503064e9e9'}, 'oper': 'CREATE', 'request-id': 'req-bb6fa775-a0ca-48ca-8b64-5f578ab02999', 'type': 'firewall_rule', 'uuid': 'f1827da2-8bae-4c45-87e0-fa503064e9e9'} 09/01/2025 01:35:09.205 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-e7805a5b-0948-4ef5-801c-53f81f0cf097', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:09.389 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-kube-node-lease-egress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-kube-node-lease-egress', 'endpoint_1': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842755], 'tags': ['namespace=kube-node-lease'], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-kube-node-lease-egress'], 'id_perms': {'created': '2025-09-01T01:35:09.334663', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:09.334663', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11673476119293380291, 'uuid_mslong': 16111642645617723340}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=kube-node-lease'], 'uuid': '9ca94da1-8880-403e-b57f-a1d4bd60ebef'}], 'uuid': 'df980d9f-8014-47cc-a200-84ae887cc6c3'}, 'oper': 'CREATE', 'request-id': 'req-4ab8947c-05ac-452c-af37-fcdf76759d32', 'type': 'firewall_rule', 'uuid': 'df980d9f-8014-47cc-a200-84ae887cc6c3'} 09/01/2025 01:35:09.498 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-9e6d47a6-1950-4eec-81b8-2065014e081b', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:09.500 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=5 timeout=120) 09/01/2025 01:35:09.500 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=kube-public event_type=ADDED kind=Namespace ns=None) 09/01/2025 01:35:09.500 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got ADDED Namespace kube-public:4422e8b3-382f-457d-8515-b4e0ab6cae7b 09/01/2025 01:35:09.533 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kubernetes.io__slash__metadata.name=kube-public'], 'obj_dict': {'display_name': 'kubernetes.io__slash__metadata.name=kube-public', 'fq_name': ['kubernetes.io__slash__metadata.name=kube-public'], 'id_perms': {'created': '2025-09-01T01:35:09.525668', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:09.525668', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12857215920637343278, 'uuid_mslong': 6157200344163633711}}, 'name': 'kubernetes.io__slash__metadata.name=kube-public', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01020003', 'tag_type_name': 'kubernetes.io__slash__metadata.name', 'tag_type_refs': [{'to': ['kubernetes.io__slash__metadata.name'], 'uuid': 'c4c51d7c-32fa-45a0-bfe9-26e1923cb4f6'}], 'tag_value': 'kube-public', 'uuid': '5572c517-3cc9-4a2f-b26e-01c26615f62e'}, 'oper': 'CREATE', 'request-id': 'req-4378319e-3e3b-4973-b6fa-04b78261c136', 'type': 'tag', 'uuid': '5572c517-3cc9-4a2f-b26e-01c26615f62e'} 09/01/2025 01:35:09.627 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['namespace=kube-public'], 'obj_dict': {'display_name': 'namespace=kube-public', 'fq_name': ['namespace=kube-public'], 'id_perms': {'created': '2025-09-01T01:35:09.612231', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:09.612231', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11728912746587169791, 'uuid_mslong': 17558590392535762952}}, 'name': 'namespace=kube-public', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01010004', 'tag_type_name': 'namespace', 'tag_type_refs': [{'to': ['namespace'], 'uuid': '884e5052-64fc-44a5-a883-82d91352fa8e'}], 'tag_value': 'kube-public', 'uuid': 'f3aca4ee-bfae-4808-a2c5-780025e123ff'}, 'oper': 'CREATE', 'request-id': 'req-ca3d8c77-1039-413f-9d2b-b53c4a26afe7', 'type': 'tag', 'uuid': 'f3aca4ee-bfae-4808-a2c5-780025e123ff'} 09/01/2025 01:35:09.714 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public'], 'obj_dict': {'display_name': 'k8s-kube-public', 'fq_name': ['default-domain', 'k8s-kube-public'], 'id_perms': {'created': '2025-09-01T01:35:09.709338', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:09.709338', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12382982719713215992, 'uuid_mslong': 1059935213013714696}}, 'parent_type': 'domain', 'parent_uuid': 'ce056a4b-12ea-41a5-896d-393d506f2011', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '0eb5a575-e09c-4708-abd9-312cfc4e85f8'}, 'oper': 'CREATE', 'request-id': 'req-69797a7b-9f8e-4b7f-a6af-3652d2ea5c0d', 'type': 'project', 'uuid': '0eb5a575-e09c-4708-abd9-312cfc4e85f8'} 09/01/2025 01:35:09.750 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'display_name': 'default-application-policy-set', 'fq_name': ['default-domain', 'k8s-kube-public', 'default-application-policy-set'], 'id_perms': {'created': '2025-09-01T01:35:09.744535', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:09.744535', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'contrail-api', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9582523444762837509, 'uuid_mslong': 5637291694719647816}}, 'parent_type': 'project', 'parent_uuid': '0eb5a575-e09c-4708-abd9-312cfc4e85f8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '4e3baeed-bf06-4048-84fb-f64d78f10205'}, 'oper': 'CREATE', 'request-id': 'req-69797a7b-9f8e-4b7f-a6af-3652d2ea5c0d', 'type': 'application_policy_set', 'uuid': '4e3baeed-bf06-4048-84fb-f64d78f10205'} 09/01/2025 01:35:09.770 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public'], 'oper': 'UPDATE', 'request-id': 'req-69797a7b-9f8e-4b7f-a6af-3652d2ea5c0d', 'type': 'project', 'uuid': '0eb5a575-e09c-4708-abd9-312cfc4e85f8'} 09/01/2025 01:35:09.862 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public', 'default'], 'obj_dict': {'display_name': 'default', 'fq_name': ['default-domain', 'k8s-kube-public', 'default'], 'id_perms': {'created': '2025-09-01T01:35:09.803936', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:35:09.803936', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10347208463162891061, 'uuid_mslong': 4265717819942062928}}, 'parent_type': 'project', 'parent_uuid': '0eb5a575-e09c-4708-abd9-312cfc4e85f8', 'perms2': {'global_access': 0, 'owner': '0eb5a575-e09c-4708-abd9-312cfc4e85f8', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': [{'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '89a68178-7737-4108-80a0-4986bfcd3106', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-kube-public:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '9c3b6f70-ae91-47c8-9df3-67b2252ae2e5', 'src_addresses': [{'network_policy': None, 'security_group': 'default-domain:k8s-kube-public:default', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '67b96713-a84f-4494-b1c6-c2c2977ce9aa', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '84c84276-8154-458c-8198-d0d82912d3e2', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}]}, 'security_group_id': 8000010, 'uuid': '3b32dfc1-a9b3-4b50-8f98-ab467fbd5b35'}, 'oper': 'CREATE', 'request-id': 'req-5b85b15d-7427-4962-8c7d-ac5d9951d4dc', 'type': 'security_group', 'uuid': '3b32dfc1-a9b3-4b50-8f98-ab467fbd5b35'} 09/01/2025 01:35:09.964 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public', 'default', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000010', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '89a68178-7737-4108-80a0-4986bfcd3106'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': '8000010', 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '9c3b6f70-ae91-47c8-9df3-67b2252ae2e5'}], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-public', 'default', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:09.901575', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:09.901575', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10631142203170786979, 'uuid_mslong': 10153433766146494016}}, 'parent_type': 'security-group', 'parent_uuid': '3b32dfc1-a9b3-4b50-8f98-ab467fbd5b35', 'perms2': {'global_access': 0, 'owner': '0eb5a575-e09c-4708-abd9-312cfc4e85f8', 'owner_access': 7, 'share': []}, 'uuid': '8ce83e37-119e-4e40-9389-6781f0a88ea3'}, 'oper': 'CREATE', 'request-id': 'req-b2df4bdd-510a-4cec-859a-569a47bede46', 'type': 'access_control_list', 'uuid': '8ce83e37-119e-4e40-9389-6781f0a88ea3'} 09/01/2025 01:35:09.972 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public'], 'oper': 'UPDATE', 'request-id': 'req-0d9c0810-b84e-4dc7-8de0-7749d2662d96', 'type': 'project', 'uuid': '0eb5a575-e09c-4708-abd9-312cfc4e85f8'} 09/01/2025 01:35:10.071 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-336387db-eb60-4c6f-b119-859f4528e9b5', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:35:10.140 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public', 'default', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '67b96713-a84f-4494-b1c6-c2c2977ce9aa'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '84c84276-8154-458c-8198-d0d82912d3e2'}], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-public', 'default', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:10.087967', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:10.087967', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13567071936738506524, 'uuid_mslong': 10770495866812713647}}, 'parent_type': 'security-group', 'parent_uuid': '3b32dfc1-a9b3-4b50-8f98-ab467fbd5b35', 'perms2': {'global_access': 0, 'owner': '0eb5a575-e09c-4708-abd9-312cfc4e85f8', 'owner_access': 7, 'share': []}, 'uuid': '95787ced-242f-4aaf-bc47-ec0b8778371c'}, 'oper': 'CREATE', 'request-id': 'req-10689d71-6487-4a8e-8116-c61e7f4b0245', 'type': 'access_control_list', 'uuid': '95787ced-242f-4aaf-bc47-ec0b8778371c'} 09/01/2025 01:35:10.149 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-336387db-eb60-4c6f-b119-859f4528e9b5', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:35:10.206 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-0e66adc2-242b-42f1-9769-0e79d035d3a3', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:35:10.230 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-0e66adc2-242b-42f1-9769-0e79d035d3a3', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:35:10.320 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public', 'k8s-kube-public-default-sg'], 'obj_dict': {'display_name': 'k8s-kube-public-default-sg', 'fq_name': ['default-domain', 'k8s-kube-public', 'k8s-kube-public-default-sg'], 'id_perms': {'created': '2025-09-01T01:35:10.267151', 'creator': None, 'description': 'Default security group', 'enable': True, 'last_modified': '2025-09-01T01:35:10.267151', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12841608011934621753, 'uuid_mslong': 4609916113583950431}}, 'parent_type': 'project', 'parent_uuid': '0eb5a575-e09c-4708-abd9-312cfc4e85f8', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': [{'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '14e9c1ba-6529-4291-bf18-e7857b87aab3', 'src_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '1afc35ea-9414-4d93-8c2b-028e2d41e118', 'src_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv4', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': 'c2ea3148-20d1-4715-9670-bc1f2d678de9', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}, {'action_list': None, 'application': [], 'created': None, 'description': None, 'direction': '>', 'dst_addresses': [{'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}], 'dst_ports': [{'end_port': 65535, 'start_port': 0}], 'ethertype': 'IPv6', 'last_modified': None, 'protocol': 'any', 'rule_sequence': None, 'rule_uuid': '84d3d5d3-f956-4e38-a9e1-e3f1b77f0521', 'src_addresses': [{'network_policy': None, 'security_group': 'local', 'subnet': None, 'subnet_list': [], 'virtual_network': None}], 'src_ports': [{'end_port': 65535, 'start_port': 0}]}]}, 'security_group_id': 8000011, 'uuid': '3ff9b647-ea85-4a5f-b236-8e72f82c6039'}, 'oper': 'CREATE', 'request-id': 'req-22391dd1-b0ec-43b0-bc2b-a3ce20ed6755', 'type': 'security_group', 'uuid': '3ff9b647-ea85-4a5f-b236-8e72f82c6039'} 09/01/2025 01:35:10.347 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_security_group_id': None}, 'fq_name': ['default-domain', 'k8s-kube-public', 'k8s-kube-public-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-7f54334e-d06d-4159-923c-4e9e6e78bcd1', 'type': 'security_group', 'uuid': '3ff9b647-ea85-4a5f-b236-8e72f82c6039'} 09/01/2025 01:35:10.365 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public', 'k8s-kube-public-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-d20e9dc9-64db-45da-8c5e-3d79e8f31f69', 'type': 'security_group', 'uuid': '3ff9b647-ea85-4a5f-b236-8e72f82c6039'} 09/01/2025 01:35:10.458 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public', 'k8s-kube-public-default-sg', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '14e9c1ba-6529-4291-bf18-e7857b87aab3'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '1afc35ea-9414-4d93-8c2b-028e2d41e118'}], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-public', 'k8s-kube-public-default-sg', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:10.386974', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:10.386974', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13390437058598049228, 'uuid_mslong': 5502608932187098816}}, 'parent_type': 'security-group', 'parent_uuid': '3ff9b647-ea85-4a5f-b236-8e72f82c6039', 'perms2': {'global_access': 0, 'owner': 'None', 'owner_access': 7, 'share': []}, 'uuid': '4c5d31ab-74e7-4ec0-b9d4-638f28f695cc'}, 'oper': 'CREATE', 'request-id': 'req-e64d83b6-7531-49c9-ba60-e2c61eb819a2', 'type': 'access_control_list', 'uuid': '4c5d31ab-74e7-4ec0-b9d4-638f28f695cc'} 09/01/2025 01:35:10.548 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-public', 'k8s-kube-public-default-sg', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv4', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': 'c2ea3148-20d1-4715-9670-bc1f2d678de9'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': {'ip_prefix': '::', 'ip_prefix_len': 0}, 'subnet_list': [], 'virtual_network': None}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': 'IPv6', 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': None}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '84d3d5d3-f956-4e38-a9e1-e3f1b77f0521'}], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'k8s-kube-public', 'k8s-kube-public-default-sg', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-01T01:35:10.498098', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:10.498098', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13038786667139761078, 'uuid_mslong': 14716788366151010158}}, 'parent_type': 'security-group', 'parent_uuid': '3ff9b647-ea85-4a5f-b236-8e72f82c6039', 'perms2': {'global_access': 0, 'owner': '0eb5a575-e09c-4708-abd9-312cfc4e85f8', 'owner_access': 7, 'share': []}, 'uuid': 'cc3c890b-cf08-476e-b4f3-13642cf703b6'}, 'oper': 'CREATE', 'request-id': 'req-70d96687-406b-47d9-919b-c0e5eb49f2de', 'type': 'access_control_list', 'uuid': 'cc3c890b-cf08-476e-b4f3-13642cf703b6'} 09/01/2025 01:35:10.768 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-kube-public-ingress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-kube-public-ingress', 'endpoint_1': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842756], 'tags': ['namespace=kube-public'], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-kube-public-ingress'], 'id_perms': {'created': '2025-09-01T01:35:10.711118', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:10.711118', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10607826179290505792, 'uuid_mslong': 13188655992088249721}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=kube-public'], 'uuid': 'f3aca4ee-bfae-4808-a2c5-780025e123ff'}], 'uuid': 'b70784bf-913f-4579-9336-91b517b32240'}, 'oper': 'CREATE', 'request-id': 'req-49b6e189-d5b9-4351-9c22-d694d7476369', 'type': 'firewall_rule', 'uuid': 'b70784bf-913f-4579-9336-91b517b32240'} 09/01/2025 01:35:10.882 7f490d10c040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:10.883 7f490d10c040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:10.883 7f490d10c040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:10.883 7f490d10c040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:10.883 7f490d10c040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:10.883 7f490d10c040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:10.883 7f490d10c040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:10.884 7f490d10c040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:10.884 7f490d10c040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:10.884 7f490d10c040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:10.884 7f490d10c040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:10.884 7f490d10c040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:10.884 7f490d10c040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:10.884 7f490d10c040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:10.889 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-d3290e7e-7c81-444e-8025-4446899997ac', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:11.087 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-kube-public-egress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-kube-public-egress', 'endpoint_1': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842756], 'tags': ['namespace=kube-public'], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-kube-public-egress'], 'id_perms': {'created': '2025-09-01T01:35:11.034553', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:11.034553', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12687764789551225607, 'uuid_mslong': 7321813596295873554}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=kube-public'], 'uuid': 'f3aca4ee-bfae-4808-a2c5-780025e123ff'}], 'uuid': '659c4eac-56d7-4412-b013-feda7ae8db07'}, 'oper': 'CREATE', 'request-id': 'req-d04bb5ab-e8db-4ce0-9dcb-da64fd7f9641', 'type': 'firewall_rule', 'uuid': '659c4eac-56d7-4412-b013-feda7ae8db07'} 09/01/2025 01:35:11.243 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=4 timeout=120) 09/01/2025 01:35:11.254 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=kube-system event_type=ADDED kind=Namespace ns=None) 09/01/2025 01:35:11.254 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got ADDED Namespace kube-system:67c6dbb9-5906-4970-985c-40b5193f4d8a 09/01/2025 01:35:11.256 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-c9cfc777-6c6e-4025-a0f0-2046f5cc631e', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:11.294 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['kubernetes.io__slash__metadata.name=kube-system'], 'obj_dict': {'display_name': 'kubernetes.io__slash__metadata.name=kube-system', 'fq_name': ['kubernetes.io__slash__metadata.name=kube-system'], 'id_perms': {'created': '2025-09-01T01:35:11.284227', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:11.284227', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13811631599716858853, 'uuid_mslong': 8548833652210091014}}, 'name': 'kubernetes.io__slash__metadata.name=kube-system', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_id': '0x01020004', 'tag_type_name': 'kubernetes.io__slash__metadata.name', 'tag_type_refs': [{'to': ['kubernetes.io__slash__metadata.name'], 'uuid': 'c4c51d7c-32fa-45a0-bfe9-26e1923cb4f6'}], 'tag_value': 'kube-system', 'uuid': '76a38ee9-bbe3-4406-bfac-c5c345aefbe5'}, 'oper': 'CREATE', 'request-id': 'req-4b8f40b3-4cd2-4cca-8a4d-9bb8bf2211f4', 'type': 'tag', 'uuid': '76a38ee9-bbe3-4406-bfac-c5c345aefbe5'} 09/01/2025 01:35:11.500 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system'], 'oper': 'UPDATE', 'request-id': 'req-f03f9d16-99fd-49ac-830d-989a5482057c', 'type': 'project', 'uuid': 'a5dc0037-9655-44b1-8af6-f013fea66ee0'} 09/01/2025 01:35:11.593 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-1e81e314-b72a-41a5-9d7a-1d919389fb88', 'type': 'virtual_network', 'uuid': 'd650e01f-39d4-4eb1-91f8-ca54373fcba6'} 09/01/2025 01:35:11.616 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-pod-network', 'k8s-default-pod-network'], 'oper': 'UPDATE', 'request-id': 'req-1e81e314-b72a-41a5-9d7a-1d919389fb88', 'type': 'routing_instance', 'uuid': '5fa5f383-4684-4ec5-88cc-18491581bc2e'} 09/01/2025 01:35:11.669 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_vxlan_network_identifier': None}, 'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-e53f458c-8756-4e94-bbf6-1fa599f7e4b4', 'type': 'virtual_network', 'uuid': 'a6c41605-29b5-4568-a40b-17e3f103c269'} 09/01/2025 01:35:11.696 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-default', 'k8s-default-service-network', 'k8s-default-service-network'], 'oper': 'UPDATE', 'request-id': 'req-e53f458c-8756-4e94-bbf6-1fa599f7e4b4', 'type': 'routing_instance', 'uuid': 'a099888b-9d11-4587-8b98-4d1673309c9e'} 09/01/2025 01:35:11.819 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_security_group_id': None}, 'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-c5c73fdb-c44a-450f-a44a-0ad4bdee763f', 'type': 'security_group', 'uuid': 'e0fa3e04-8145-47f6-9b69-718272132295'} 09/01/2025 01:35:11.852 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'extra_dict': {'deallocated_security_group_id': None}, 'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg'], 'oper': 'UPDATE', 'request-id': 'req-f4377f5a-1796-4fa8-b1ea-fb62420a7ca8', 'type': 'security_group', 'uuid': 'e0fa3e04-8145-47f6-9b69-718272132295'} 09/01/2025 01:35:11.864 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-f66c4fd0-b770-4d76-99e7-c8fd6bbfe945', 'type': 'access_control_list', 'uuid': '299bf655-b7a5-41e2-877c-35bccfef2eda'} 09/01/2025 01:35:11.880 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'k8s-kube-system-default-sg', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-1b59da51-eaa3-4e2f-8896-b8802e3a3132', 'type': 'access_control_list', 'uuid': '448ddfc1-0480-4786-9b61-9816732b043b'} 09/01/2025 01:35:12.057 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-kube-system-ingress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-kube-system-ingress', 'endpoint_1': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842752], 'tags': ['namespace=kube-system'], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-kube-system-ingress'], 'id_perms': {'created': '2025-09-01T01:35:12.005815', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:12.005815', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12917800681854254437, 'uuid_mslong': 9121708868258582194}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=kube-system'], 'uuid': '317d6be2-b0aa-4cee-87d2-56074b8ba3b1'}], 'uuid': '7e96d1e3-0515-4eb2-b345-3f475938bd65'}, 'oper': 'CREATE', 'request-id': 'req-a571d2ec-12f7-40d9-a141-8aa030568bcc', 'type': 'firewall_rule', 'uuid': '7e96d1e3-0515-4eb2-b345-3f475938bd65'} 09/01/2025 01:35:12.073 7f490d0ed540 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:35:12.196 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-cd5f9f37-b7fe-4e40-bd9d-4d0cf7581682', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:12.533 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-Namespace-kube-system-egress'], 'obj_dict': {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': '>', 'display_name': 'k8s-Namespace-kube-system-egress', 'endpoint_1': {'address_group': None, 'any': None, 'subnet': None, 'tag_ids': [16842752], 'tags': ['namespace=kube-system'], 'virtual_network': None}, 'endpoint_2': {'address_group': None, 'any': True, 'subnet': None, 'tag_ids': [], 'tags': [], 'virtual_network': None}, 'fq_name': ['default-policy-management', 'k8s-Namespace-kube-system-egress'], 'id_perms': {'created': '2025-09-01T01:35:12.470619', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:12.470619', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11622934083351091608, 'uuid_mslong': 14421497729005404891}}, 'match_tag_types': {'tag_type': [1]}, 'parent_type': 'policy-management', 'parent_uuid': '4a4fd292-14a2-4148-9861-3579ce70f727', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service': {'dst_ports': {'end_port': 65535, 'start_port': 0}, 'protocol': 'any', 'protocol_id': 0, 'src_ports': None}, 'tag_refs': [{'to': ['namespace=kube-system'], 'uuid': '317d6be2-b0aa-4cee-87d2-56074b8ba3b1'}], 'uuid': 'c82373c6-b278-42db-a14c-f4f7c5ec1998'}, 'oper': 'CREATE', 'request-id': 'req-fa8d65a1-5ce9-40f7-90e7-60e71d67ac20', 'type': 'firewall_rule', 'uuid': 'c82373c6-b278-42db-a14c-f4f7c5ec1998'} 09/01/2025 01:35:12.664 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-policy-management', 'k8s-allowall'], 'oper': 'UPDATE', 'request-id': 'req-cc473e95-bceb-4aee-a9bb-72ffb67a594f', 'type': 'firewall_policy', 'uuid': 'aebd7f43-362b-4543-a3a3-c56771bd18d7'} 09/01/2025 01:35:12.703 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=3 timeout=120) 09/01/2025 01:35:12.703 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:35:12.703 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:35:12.703 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:35:12.703 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=2 timeout=120) 09/01/2025 01:35:12.704 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:35:12.704 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:35:12.704 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:35:12.704 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=coredns-6f867cd986-wlkqx event_type=MODIFIED kind=Pod ns=kube-system) 09/01/2025 01:35:12.704 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncPod - Got MODIFIED Pod kube-system:coredns-6f867cd986-wlkqx:a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8 09/01/2025 01:35:12.775 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncPod - Vrouter 10.0.0.22 (pod_node=an-jenkins-deploy-platform-juju-k8s-1454-1) Not Found for Pod a4a425f1-8708-4c1c-b41f-cd30b9d1d8a8: routers=[] 09/01/2025 01:35:12.809 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:35:12.817 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-domain', 'k8s-kube-system', 'coredns-6f867cd986-wlkqx__e02048c4-86d3-11f0-93e1-fa163e599e0d'], 'oper': 'UPDATE', 'request-id': 'req-fd135d25-ba94-42ee-a11e-0478c91d7bdf', 'type': 'virtual_machine_interface', 'uuid': 'e02048c4-86d3-11f0-93e1-fa163e599e0d'} 09/01/2025 01:35:14.884 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:14.886 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:14.886 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:14.886 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:14.886 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:14.886 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:14.886 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:14.886 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:14.887 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:14.887 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:14.887 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:14.887 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:14.887 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:14.887 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:18.886 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:18.887 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:18.887 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:18.887 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:18.887 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:18.888 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:22.888 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:22.889 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:22.889 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:22.889 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:22.890 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:26.890 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:26.891 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:26.891 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:26.891 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:26.891 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:26.891 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:26.891 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:26.891 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:26.891 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:26.892 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:26.892 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:26.892 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:26.892 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:26.892 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:30.892 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:30.893 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:30.894 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:30.894 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:30.894 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:34.893 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:34.895 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:34.895 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:34.895 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:34.895 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:34.895 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:34.895 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:34.896 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:34.896 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:34.896 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:34.896 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:34.896 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:34.896 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:34.896 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:38.896 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:38.898 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:38.898 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:38.898 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:38.898 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:38.898 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:38.898 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:38.899 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:38.899 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:38.899 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:38.899 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:38.899 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:38.899 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:38.899 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:42.898 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:42.899 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:42.899 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:42.899 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:42.900 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:46.901 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:46.902 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:46.902 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:46.902 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:46.903 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:46.903 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:46.903 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:46.903 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:46.903 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:46.903 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:46.903 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:46.903 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:46.904 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:46.904 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:49.501 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-5c9a772d-a744-45b7-80ef-09a53d7ad23d', 'type': 'global_system_config', 'uuid': '70eaa5ad-608f-40c9-a193-2be391842efd'} 09/01/2025 01:35:49.574 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-juju-k8s-1454-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-juju-k8s-1454-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-juju-k8s-1454-1'], 'id_perms': {'created': '2025-09-01T01:35:49.567966', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:49.567966', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10794658036228347919, 'uuid_mslong': 12163681958422269707}}, 'parent_type': 'global-system-config', 'parent_uuid': '70eaa5ad-608f-40c9-a193-2be391842efd', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a8ce144b-9496-4f0b-95ce-544a9934d80f', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.0.0.22'}, 'oper': 'CREATE', 'request-id': 'req-ee08329c-5872-4f7a-b645-2f27e34a25a4', 'type': 'virtual_router', 'uuid': 'a8ce144b-9496-4f0b-95ce-544a9934d80f'} 09/01/2025 01:35:49.681 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-juju-k8s-1454-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-juju-k8s-1454-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T01:35:49.668311', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T01:35:49.668311', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12472940248073749293, 'uuid_mslong': 15418220656626518203}}, 'parent_type': 'virtual-router', 'parent_uuid': 'a8ce144b-9496-4f0b-95ce-544a9934d80f', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd5f885ff-7ad0-48bb-ad18-c912371bc32d', 'virtual_machine_interface_disable_policy': False, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:d5:f8:85:ff:7a']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'e07e6e04-4e93-4043-ab83-32b8f8854bbe'}]}, 'oper': 'CREATE', 'request-id': 'req-f6bff454-c5c2-4c20-b305-c5e32915e30b', 'type': 'virtual_machine_interface', 'uuid': 'd5f885ff-7ad0-48bb-ad18-c912371bc32d'} 09/01/2025 01:35:49.703 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-juju-k8s-1454-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-f6bff454-c5c2-4c20-b305-c5e32915e30b', 'type': 'virtual_machine_interface', 'uuid': 'd5f885ff-7ad0-48bb-ad18-c912371bc32d'} 09/01/2025 01:35:50.904 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:50.905 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:50.905 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:50.905 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:50.905 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:50.905 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:50.905 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:50.905 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:50.905 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:50.906 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:50.906 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:50.906 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:50.906 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:50.906 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:54.908 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:54.909 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:54.909 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:54.909 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:54.909 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:54.909 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:54.909 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:54.909 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:54.909 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:54.910 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:54.910 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:54.910 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:54.910 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:54.910 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:58.909 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:35:58.911 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:35:58.911 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:35:58.911 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:58.911 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:35:58.911 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:35:58.911 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:35:58.911 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:35:58.912 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:58.912 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:35:58.912 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:58.912 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:58.912 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:35:58.912 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:35:59.132 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=381 newResVer=388 09/01/2025 01:35:59.132 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:35:59.132 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:35:59.132 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:36:02.912 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:02.913 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:02.913 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:02.913 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:02.913 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:02.913 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:02.914 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:02.914 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:02.914 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:02.914 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:02.914 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:02.914 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:02.914 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:02.914 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:06.914 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:06.915 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:06.915 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:06.915 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:06.916 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:10.916 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:10.917 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:10.917 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:10.917 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:10.917 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:10.917 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:10.917 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:10.918 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:10.918 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:10.918 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:10.918 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:10.918 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:10.918 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:10.918 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:14.918 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:14.919 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:14.919 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:14.919 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:14.919 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:14.919 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:14.919 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:14.920 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:14.920 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:14.920 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:14.920 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:14.920 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:14.920 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:14.920 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:18.920 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:18.921 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:18.922 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:18.923 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:18.923 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:22.922 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:22.923 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:22.923 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:22.923 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:22.923 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:22.923 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:22.923 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:22.924 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:22.924 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:22.924 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:22.924 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:22.924 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:22.924 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:22.924 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:26.925 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:26.926 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:26.926 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:26.926 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:26.926 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:26.926 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:26.926 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:26.927 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:26.927 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:26.927 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:26.927 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:26.927 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:26.927 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:26.927 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:30.927 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:30.928 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:30.929 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:30.929 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:30.929 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:30.929 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:30.929 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:30.929 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:30.929 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:30.930 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:30.930 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:30.930 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:30.930 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:30.930 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:34.929 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:34.930 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:34.930 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:34.930 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:34.930 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:34.930 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:34.930 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:34.931 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:34.931 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:34.931 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:34.931 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:34.931 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:34.931 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:34.931 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:38.931 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:38.932 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:38.932 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:38.933 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:40.628 7f490d155400 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:36:42.935 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:42.936 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:42.936 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:42.936 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:42.937 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:46.937 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:46.938 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:46.938 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:46.938 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:46.938 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:46.938 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:46.938 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:46.939 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:46.939 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:46.939 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:46.939 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:46.939 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:46.939 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:46.939 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:50.940 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:50.941 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:50.941 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:50.941 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:50.941 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:50.941 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:50.941 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:50.942 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:50.942 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:50.942 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:50.942 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:50.942 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:50.942 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:50.942 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:54.942 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:54.943 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:54.943 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:54.943 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:54.943 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:54.943 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:54.943 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:54.943 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:54.943 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:54.944 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:54.944 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:54.944 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:54.944 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:54.944 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:58.944 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:36:58.945 7f490d155400 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:36:58.945 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:36:58.945 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:58.945 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:36:58.945 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:36:58.946 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:36:58.946 7f490d155400 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:36:58.946 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:58.946 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:36:58.946 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:58.946 7f490d155400 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:58.946 7f490d155400 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:36:58.946 7f490d155400 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:36:59.623 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=388 09/01/2025 01:36:59.623 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:36:59.623 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:36:59.623 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:36:59.880 7f490d079ae0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.880 7f490d079ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.880 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:36:59.881 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:36:59.881 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:36:59.881 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 01:36:59.881 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:36:59.881 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:36:59.882 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:36:59.883 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:36:59.883 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:36:59.883 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:36:59.887 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.888 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.888 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:36:59.888 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:36:59.888 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:36:59.888 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:36:59.888 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:36:59.888 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:36:59.889 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:36:59.889 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:36:59.891 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:36:59.891 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:36:59.897 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:36:59.898 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:36:59.902 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.902 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.902 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:36:59.902 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:36:59.902 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:36:59.903 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:36:59.903 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:36:59.903 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:36:59.904 7f490d0224a0 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.904 7f490d0224a0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.904 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:36:59.904 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:36:59.904 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:36:59.904 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:36:59.904 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:36:59.904 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:36:59.905 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 01:36:59.905 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:36:59.905 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:36:59.906 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:36:59.906 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:36:59.906 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:36:59.906 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:36:59.910 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:36:59.911 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:36:59.918 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.918 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.918 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:36:59.918 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:36:59.918 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:36:59.918 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:36:59.918 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:36:59.919 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:36:59.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:36:59.920 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:36:59.920 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:36:59.920 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:36:59.923 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:36:59.954 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.954 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:36:59.954 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:36:59.954 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=385 09/01/2025 01:36:59.954 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:36:59.954 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:36:59.955 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:36:59.955 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'})(timeout=120) 09/01/2025 01:36:59.956 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:36:59.956 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:36:59.956 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:36:59.960 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'}) 09/01/2025 01:37:02.947 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:03.007 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:03.007 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:03.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:07.008 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:07.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:07.010 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:07.010 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:11.009 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:11.010 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:11.010 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:11.011 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:15.013 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:15.014 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:15.014 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:15.014 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:15.015 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:15.015 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:15.015 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:15.015 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:15.015 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:15.015 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:15.015 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:15.015 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:15.016 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:15.016 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:19.017 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:19.018 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:19.018 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:19.018 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:19.018 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:19.018 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:19.018 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:19.018 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:19.019 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:19.019 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:19.019 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:19.019 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:19.019 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:19.019 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:23.018 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:23.019 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:23.019 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:23.019 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:23.020 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:26.277 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-juju-k8s-1454-1'], 'oper': 'UPDATE', 'request-id': 'req-289c98ce-ddb2-46fa-a5f5-966a1b563ad8', 'type': 'virtual_router', 'uuid': 'a8ce144b-9496-4f0b-95ce-544a9934d80f'} 09/01/2025 01:37:26.345 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-juju-k8s-1454-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-b7f2219d-1994-43ae-91a0-48b4f5976fda', 'type': 'virtual_machine_interface', 'uuid': 'd5f885ff-7ad0-48bb-ad18-c912371bc32d'} 09/01/2025 01:37:27.020 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:27.021 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:27.021 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:27.021 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:27.021 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:27.022 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:27.022 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:27.022 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:27.022 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:27.022 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:27.022 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:27.022 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:27.022 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:27.023 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:31.023 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:31.024 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:31.025 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:31.026 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:31.026 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:31.026 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:35.026 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:35.027 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:35.027 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:35.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:39.028 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:39.030 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:39.031 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:39.031 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:39.031 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:39.031 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:43.031 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:43.032 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:43.032 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:43.032 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:43.032 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:43.032 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:43.032 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:43.033 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:43.033 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:43.033 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:43.033 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:43.033 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:43.033 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:43.033 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:47.033 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:47.034 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:47.034 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:47.034 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:47.034 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:47.034 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:47.034 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:47.035 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:47.035 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:47.035 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:47.035 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:47.035 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:47.035 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:47.035 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:51.035 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:51.036 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:51.036 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:51.037 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:55.038 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:55.040 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:55.040 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:55.040 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:55.040 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:55.040 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:55.040 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:55.040 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:55.041 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:55.041 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:55.041 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:55.041 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:55.041 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:55.041 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:55.475 7f490d10ca40 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:37:57.841 7f490d10ca40 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:37:59.040 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:37:59.041 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:37:59.041 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:37:59.042 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:37:59.135 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=388 09/01/2025 01:37:59.135 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:37:59.135 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:37:59.135 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:38:01.331 7f490d0ed040 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:38:03.042 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:03.044 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:03.044 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:03.044 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:03.044 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:03.044 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:03.044 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:03.044 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:03.044 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:03.045 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:03.045 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:03.045 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:03.045 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:03.045 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:07.046 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:07.047 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:07.048 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:07.048 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:07.048 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:07.048 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:11.050 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:11.051 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:11.051 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:11.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:15.052 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:15.053 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:15.053 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:15.054 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:15.055 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:19.055 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:19.056 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:19.056 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:19.057 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:23.058 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:23.059 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:23.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:27.060 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:27.061 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:27.062 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:27.063 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:27.063 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:27.063 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:31.062 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:31.063 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:31.064 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:35.067 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:35.068 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:35.068 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:35.068 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:35.068 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:35.068 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:35.068 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:35.069 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:35.069 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:35.069 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:35.069 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:35.069 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:35.069 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:35.069 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:39.068 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:39.069 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:39.069 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:39.069 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:39.070 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:43.070 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:43.072 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:43.072 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:43.072 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:43.073 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:47.075 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:47.076 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:47.076 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:47.076 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:47.076 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:47.076 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:47.076 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:47.077 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:47.077 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:47.077 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:47.077 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:47.077 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:47.077 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:47.077 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:51.077 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:51.078 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:51.078 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:51.079 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:55.081 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:55.092 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:55.092 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:55.092 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:55.093 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:59.094 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:38:59.095 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:38:59.095 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:38:59.096 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:38:59.881 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=388 09/01/2025 01:38:59.881 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:38:59.881 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:38:59.881 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:38:59.898 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.898 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.899 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:38:59.899 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:38:59.899 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:38:59.899 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:38:59.899 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:38:59.900 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:38:59.901 7f490d079ae0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.902 7f490d079ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.902 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:38:59.902 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:38:59.903 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:38:59.903 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:38:59.903 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:38:59.903 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 01:38:59.903 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:38:59.904 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:38:59.906 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:38:59.906 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:38:59.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:38:59.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:38:59.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:38:59.907 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:38:59.911 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.912 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.912 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:38:59.912 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:38:59.912 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:38:59.912 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:38:59.912 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:38:59.912 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:38:59.915 7f490d0224a0 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.915 7f490d0224a0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.915 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:38:59.916 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:38:59.916 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:38:59.916 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:38:59.916 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:38:59.916 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:38:59.916 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 01:38:59.916 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:38:59.917 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:38:59.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:38:59.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:38:59.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:38:59.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:38:59.924 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.924 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.924 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:38:59.924 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:38:59.925 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:38:59.925 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:38:59.925 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:38:59.925 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:38:59.926 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:38:59.926 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:38:59.926 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:38:59.926 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:38:59.930 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:38:59.930 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:38:59.931 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:38:59.931 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:38:59.932 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:38:59.961 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.962 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:38:59.962 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:38:59.962 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=385 09/01/2025 01:38:59.962 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:38:59.962 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:38:59.962 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:38:59.963 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'})(timeout=120) 09/01/2025 01:38:59.964 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:38:59.964 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:38:59.964 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:38:59.969 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'}) 09/01/2025 01:39:03.097 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:03.098 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:03.098 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:03.098 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:03.098 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:03.098 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:03.098 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:03.099 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:03.099 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:03.099 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:03.099 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:03.099 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:03.099 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:03.099 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:07.100 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:07.101 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:07.102 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:07.103 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:07.103 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:07.103 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:11.103 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:11.104 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:11.104 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:11.104 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:11.104 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:11.104 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:11.104 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:11.105 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:11.105 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:11.105 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:11.105 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:11.105 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:11.105 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:11.105 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:15.106 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:15.107 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:15.107 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:15.107 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:15.107 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:15.107 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:15.107 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:15.107 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:15.107 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:15.108 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:15.108 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:15.108 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:15.108 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:15.108 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:19.108 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:19.109 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:19.109 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:19.109 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:19.110 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:23.110 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:23.111 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:23.111 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:23.112 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:27.113 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:27.114 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:27.114 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:27.114 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:27.114 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:27.114 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:27.114 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:27.115 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:27.115 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:27.115 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:27.115 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:27.115 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:27.115 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:27.115 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:31.116 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:31.117 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:31.117 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:31.117 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:31.117 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:31.117 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:31.117 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:31.118 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:31.118 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:31.118 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:31.118 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:31.118 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:31.118 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:31.118 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:35.118 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:35.119 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:35.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:39.120 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:39.121 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:39.121 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:39.121 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:39.121 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:39.121 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:39.121 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:39.122 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:39.122 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:39.122 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:39.122 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:39.122 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:39.122 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:39.122 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:43.122 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:43.123 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:43.124 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:43.125 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:47.124 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:47.125 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:47.125 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:47.125 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:47.126 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:51.126 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:51.127 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:51.127 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:51.127 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:51.127 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:51.128 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:55.128 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:55.130 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:55.130 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:55.130 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:55.130 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:55.130 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:55.130 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:55.131 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:55.131 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:55.131 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:55.131 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:55.131 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:55.131 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:55.131 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:59.132 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:59.133 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:39:59.134 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:59.134 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:59.134 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:39:59.134 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:39:59.788 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=388 09/01/2025 01:39:59.789 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:39:59.789 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:39:59.789 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:39:59.789 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:39:59.789 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:39:59.790 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:39:59.804 7f490cfa6400 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-juju-k8s-1454-1'], 'oper': 'UPDATE', 'request-id': 'req-10dc3d7c-4558-47f2-a814-0dc448db550a', 'type': 'virtual_router', 'uuid': 'a8ce144b-9496-4f0b-95ce-544a9934d80f'} 09/01/2025 01:39:59.847 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:39:59.847 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:40:03.133 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:03.134 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:03.134 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:03.134 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:03.134 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:03.134 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:03.134 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:03.135 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:03.135 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:03.135 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:03.135 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:03.135 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:03.135 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:03.135 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:07.136 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:07.137 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:07.137 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:07.137 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:07.138 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:11.139 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:11.140 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:11.140 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:11.140 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:11.141 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:15.141 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:15.142 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:15.142 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:15.142 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:15.142 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:15.142 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:15.142 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:15.143 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:15.143 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:15.143 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:15.143 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:15.143 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:15.143 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:15.143 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:19.143 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:19.144 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:19.144 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:19.145 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:19.145 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:19.145 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:19.145 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:19.145 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:19.145 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:19.146 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:19.146 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:19.146 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:19.146 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:19.146 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:23.147 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:23.148 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:23.148 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:23.148 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:23.148 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:23.148 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:23.149 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:23.149 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:23.149 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:23.149 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:23.149 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:23.149 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:23.149 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:23.149 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:27.149 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:27.150 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:27.150 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:27.151 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:27.152 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:27.152 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:31.152 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:31.153 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:31.153 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:31.153 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:31.154 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:35.156 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:35.157 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:35.157 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:35.157 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:35.157 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:35.157 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:35.158 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:35.158 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:35.158 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:35.158 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:35.158 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:35.158 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:35.158 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:35.158 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:39.158 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:39.159 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:39.159 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:39.159 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:39.159 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:39.159 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:39.159 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:39.160 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:39.160 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:39.160 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:39.160 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:39.160 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:39.160 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:39.160 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:43.161 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:43.162 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:43.162 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:43.162 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:43.162 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:43.162 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:43.163 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:43.163 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:43.163 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:43.163 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:43.163 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:43.163 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:43.163 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:43.163 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:47.165 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:47.166 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:47.166 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:47.166 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:47.166 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:47.166 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:47.166 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:47.166 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:47.166 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:47.167 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:47.167 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:47.167 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:47.167 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:47.167 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:49.884 7f490d0ed360 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:40:51.167 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:51.168 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:51.168 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:51.168 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:51.168 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:51.168 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:51.168 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:51.169 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:51.169 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:51.169 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:51.169 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:51.169 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:51.169 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:51.169 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:55.169 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:55.170 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:55.170 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:55.170 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:55.170 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:55.170 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:55.170 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:55.171 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:55.171 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:55.171 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:55.171 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:55.171 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:55.171 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:55.171 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:59.173 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:40:59.174 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:40:59.174 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:40:59.174 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:59.175 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:40:59.176 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:40:59.898 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=388 09/01/2025 01:40:59.898 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:40:59.898 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:40:59.898 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:40:59.931 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.931 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.932 7f490d0224a0 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.932 7f490d0224a0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.932 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:40:59.932 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:40:59.932 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:40:59.932 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:40:59.932 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:40:59.932 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:40:59.933 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 01:40:59.933 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:40:59.933 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:40:59.934 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:40:59.934 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:40:59.934 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:40:59.935 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.935 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.936 7f490d079ae0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.936 7f490d079ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.936 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.936 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.936 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:40:59.936 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:40:59.936 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:40:59.937 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:40:59.937 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:40:59.937 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:40:59.937 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:40:59.937 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:40:59.937 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:40:59.937 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:40:59.938 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:40:59.938 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:40:59.938 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:40:59.938 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:40:59.938 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:40:59.938 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:40:59.938 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:40:59.938 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:40:59.939 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 01:40:59.939 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:40:59.940 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:40:59.940 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:40:59.940 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:40:59.941 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:40:59.943 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:40:59.943 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=3 timeout=120) 09/01/2025 01:40:59.943 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:40:59.943 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:40:59.943 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:40:59.943 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=2 timeout=120) 09/01/2025 01:40:59.943 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:40:59.944 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:40:59.944 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:40:59.944 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:40:59.944 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:40:59.944 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:40:59.944 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:40:59.956 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:40:59.956 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:40:59.957 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:40:59.957 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:40:59.958 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:40:59.970 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.970 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:40:59.971 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:40:59.971 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=385 09/01/2025 01:40:59.971 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:40:59.971 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:40:59.971 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:40:59.971 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'})(timeout=120) 09/01/2025 01:40:59.972 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:40:59.972 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:40:59.973 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:40:59.977 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'}) 09/01/2025 01:41:03.174 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:03.175 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:03.176 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:03.177 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:07.177 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:07.178 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:07.178 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:07.178 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:07.179 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:11.181 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:11.182 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:11.182 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:11.182 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:11.182 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:11.182 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:11.182 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:11.183 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:11.183 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:11.183 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:11.183 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:11.183 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:11.183 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:11.183 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:15.183 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:15.184 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:15.184 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:15.184 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:15.184 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:15.184 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:15.185 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:15.185 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:15.185 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:15.185 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:15.185 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:15.185 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:15.185 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:15.185 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:19.186 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:19.187 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:19.187 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:19.187 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:19.187 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:19.188 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:23.191 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:23.192 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:23.193 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:23.193 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:23.193 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:23.193 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:27.193 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:27.194 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:27.194 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:27.194 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:27.194 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:27.194 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:27.194 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:27.195 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:27.195 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:27.195 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:27.195 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:27.195 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:27.195 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:27.195 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:31.196 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:31.197 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:31.197 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:31.198 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:35.200 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:35.201 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:35.201 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:35.202 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:35.203 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:35.203 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:39.202 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:39.203 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:39.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:39.205 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:39.205 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:43.204 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:43.205 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:43.205 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:43.206 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:47.207 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:47.208 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:47.208 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:47.208 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:47.208 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:47.208 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:47.208 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:47.208 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:47.208 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:47.209 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:47.209 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:47.209 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:47.209 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:47.209 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:51.208 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:51.209 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:51.210 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:51.211 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:55.211 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:55.212 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:55.212 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:55.212 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:55.213 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:55.214 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:59.213 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:41:59.214 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:41:59.214 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:41:59.214 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:59.214 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:41:59.214 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:41:59.214 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:41:59.215 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:41:59.215 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:59.215 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:41:59.215 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:59.215 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:59.215 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:41:59.215 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:41:59.980 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=388 09/01/2025 01:41:59.980 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:41:59.980 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:41:59.980 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:42:03.215 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:03.217 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:03.217 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:03.217 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:03.217 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:03.217 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:03.217 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:03.217 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:03.218 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:03.218 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:03.218 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:03.218 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:03.218 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:03.218 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:07.219 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:07.220 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:07.221 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:07.221 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:11.222 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:11.223 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:11.223 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:11.223 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:11.224 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:15.224 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:15.225 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:15.225 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:15.226 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:19.227 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:19.228 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:19.228 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:19.228 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:19.228 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:19.228 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:19.228 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:19.229 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:19.229 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:19.229 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:19.229 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:19.229 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:19.229 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:19.229 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:23.229 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:23.230 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:23.230 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:23.230 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:23.230 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:23.231 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:27.231 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:27.232 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:27.232 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:27.232 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:27.232 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:27.232 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:27.232 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:27.232 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:27.233 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:27.233 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:27.233 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:27.233 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:27.233 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:27.233 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:31.233 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:31.234 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:31.235 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:31.235 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:31.235 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:31.235 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:31.235 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:31.235 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:31.235 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:31.235 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:31.236 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:31.236 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:31.236 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:31.236 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:31.692 7f490d0ed7c0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:42:34.053 7f490d0ed7c0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:42:35.235 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:35.236 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:35.236 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:35.236 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:35.236 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:35.236 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:35.236 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:35.237 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:35.237 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:35.237 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:35.237 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:35.237 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:35.237 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:35.237 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:37.655 7f490d0ed7c0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:42:39.236 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:39.237 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:39.238 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:39.239 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:39.239 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:43.240 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:43.241 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:43.242 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:43.242 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:47.242 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:47.243 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:47.244 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:51.245 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:51.246 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:51.247 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:51.247 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:51.247 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:51.247 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:55.247 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:55.248 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:55.248 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:55.248 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:55.249 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:59.250 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:42:59.252 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:42:59.252 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:42:59.252 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:59.252 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:42:59.252 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:42:59.252 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:42:59.252 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:42:59.253 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:59.253 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:42:59.253 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:59.253 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:59.253 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:42:59.253 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:42:59.386 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=388 09/01/2025 01:42:59.387 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:42:59.387 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:42:59.387 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:42:59.958 7f490d079ae0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.958 7f490d079ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.958 7f490d0224a0 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.958 7f490d0224a0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.958 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.959 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.959 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:42:59.959 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:42:59.959 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:42:59.959 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:42:59.959 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:42:59.959 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:42:59.959 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:42:59.960 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:42:59.960 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:42:59.960 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:42:59.960 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:42:59.960 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:42:59.961 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 01:42:59.961 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:42:59.962 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:42:59.963 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 01:42:59.963 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:42:59.963 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:42:59.964 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.964 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.964 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.964 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.964 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=2 timeout=120) 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:42:59.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:42:59.965 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:42:59.965 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:42:59.965 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:42:59.966 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:42:59.966 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:42:59.966 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:42:59.966 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:42:59.966 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:42:59.966 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:42:59.967 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:42:59.967 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:42:59.967 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:42:59.968 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:42:59.969 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:42:59.969 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:42:59.969 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:42:59.969 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:42:59.969 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:42:59.971 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:42:59.972 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:42:59.977 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.978 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:42:59.978 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:42:59.978 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=385 09/01/2025 01:42:59.978 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:42:59.978 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:42:59.978 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:42:59.978 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'})(timeout=120) 09/01/2025 01:42:59.979 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:42:59.979 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:42:59.980 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:42:59.983 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:42:59.984 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:42:59.984 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:42:59.985 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'}) 09/01/2025 01:42:59.985 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:42:59.986 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:43:03.253 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:03.254 7f490d0ae680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:03.254 7f490d0ae680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:03.254 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:03.254 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:03.255 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:07.256 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:07.257 7f490d0ae680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:07.257 7f490d0ae680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:07.257 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:07.257 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:07.257 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:07.257 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:07.257 7f490d0ae680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:07.257 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:07.258 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:07.258 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:07.258 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:07.258 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:07.258 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:10.099 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=388 09/01/2025 01:43:10.100 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:43:10.101 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:43:10.101 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:43:11.257 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:11.259 7f490d0edcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:11.259 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:11.259 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:11.259 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:11.260 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:11.665 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration PodMonitor 09/01/2025 01:43:11.665 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:43:11.665 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=388 09/01/2025 01:43:11.665 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:43:11.666 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 01:43:11.666 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:43:11.666 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'})(timeout=120) 09/01/2025 01:43:11.667 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:43:11.667 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:43:11.667 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:43:11.667 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:43:11.679 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'}) 09/01/2025 01:43:15.259 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:15.260 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:15.260 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:15.260 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:15.261 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:19.262 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:19.263 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:19.264 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:19.264 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:19.264 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:19.264 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:19.264 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:19.264 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:19.264 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:19.265 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:19.265 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:19.265 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:19.265 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:19.265 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:23.266 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:23.267 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:23.268 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:23.268 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:23.268 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:23.268 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:23.268 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:23.268 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:23.268 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:23.269 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:23.269 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:23.269 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:23.269 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:23.269 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:27.268 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:27.270 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:27.271 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:27.271 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:27.271 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:31.270 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:31.271 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:31.271 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:31.272 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:35.274 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:35.275 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:35.275 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:35.275 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:35.275 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:35.276 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:39.276 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:39.277 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:39.277 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:39.277 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:39.277 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:39.277 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:39.277 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:39.278 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:39.278 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:39.278 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:39.278 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:39.278 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:39.278 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:39.278 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:43.278 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:43.280 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:43.280 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:43.280 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:43.280 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:43.280 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:43.280 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:43.281 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:43.281 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:43.281 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:43.281 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:43.281 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:43.281 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:43.281 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:47.282 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:47.283 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:47.284 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:51.284 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:51.285 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:51.285 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:51.286 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:55.287 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:55.288 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:55.289 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:55.289 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:55.289 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:55.289 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:59.289 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:43:59.290 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:43:59.290 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:43:59.290 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:59.290 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:43:59.290 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:43:59.290 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:43:59.290 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:43:59.290 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:59.291 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:43:59.291 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:59.291 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:59.291 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:43:59.291 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:43:59.628 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=381 newResVer=1052 09/01/2025 01:43:59.628 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:43:59.629 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:43:59.629 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:43:59.689 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=381 newResVer=1042 09/01/2025 01:43:59.689 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:43:59.689 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:43:59.689 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:44:00.092 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=381 newResVer=1013 09/01/2025 01:44:00.092 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:44:00.093 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:44:00.093 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:44:03.290 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:03.291 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:03.292 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:03.293 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:03.293 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:07.293 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:07.294 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:07.294 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:07.294 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:07.294 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:07.294 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:07.294 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:07.295 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:07.295 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:07.295 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:07.295 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:07.295 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:07.295 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:07.295 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:11.297 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:11.298 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:11.298 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:11.298 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:11.298 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:11.298 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:11.298 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:11.299 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:11.299 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:11.299 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:11.299 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:11.299 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:11.299 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:11.299 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:15.298 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:15.299 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:15.300 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:15.301 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:19.300 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:19.301 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:19.301 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:19.301 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:19.301 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:19.301 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:19.301 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:19.302 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:19.302 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:19.302 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:19.302 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:19.302 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:19.302 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:19.302 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:23.302 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:23.303 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:23.303 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:23.303 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:23.303 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:23.303 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:23.303 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:23.303 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:23.304 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:23.304 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:23.304 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:23.304 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:23.304 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:23.304 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:27.304 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:27.305 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:27.306 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:27.306 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:27.306 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:27.306 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:31.306 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:31.307 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:31.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:35.308 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:35.309 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:35.309 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:35.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:39.310 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:39.311 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:39.311 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:39.311 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:39.311 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:39.311 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:39.311 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:39.312 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:39.312 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:39.312 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:39.312 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:39.312 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:39.312 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:39.312 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:43.313 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:43.314 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:43.314 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:43.314 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:43.314 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:43.314 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:43.314 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:43.315 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:43.315 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:43.315 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:43.315 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:43.315 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:43.315 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:43.315 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:47.314 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:47.315 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:47.315 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:47.315 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:47.316 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:51.316 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:51.317 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:51.317 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:51.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:55.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:55.319 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:55.319 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:55.319 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:55.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:59.321 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:44:59.322 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:44:59.323 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:44:59.323 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:59.323 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:44:59.323 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:44:59.323 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:44:59.323 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:44:59.323 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:59.323 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:44:59.324 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:59.324 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:59.324 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:44:59.324 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:44:59.499 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1013 newResVer=1013 09/01/2025 01:44:59.499 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:44:59.499 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:44:59.499 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:44:59.900 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1042 newResVer=1042 09/01/2025 01:44:59.901 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:44:59.901 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:44:59.901 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:44:59.954 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:44:59.954 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:44:59.954 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:44:59.955 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:44:59.984 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:44:59.984 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:44:59.984 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:44:59.985 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:44:59.985 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:44:59.985 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:44:59.985 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:44:59.985 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:44:59.987 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:44:59.987 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:44:59.987 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:44:59.987 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:44:59.987 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:44:59.987 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:44:59.987 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:44:59.987 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:44:59.987 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:44:59.988 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:44:59.988 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:44:59.988 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=385 09/01/2025 01:44:59.988 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:44:59.988 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:44:59.988 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:44:59.988 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'})(timeout=120) 09/01/2025 01:44:59.989 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:44:59.989 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:44:59.990 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:44:59.991 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:44:59.991 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:44:59.991 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:44:59.991 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:44:59.991 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:44:59.991 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:44:59.991 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:45:00.009 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '385'}) 09/01/2025 01:45:00.010 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:45:00.011 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:45:03.323 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:03.325 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:03.325 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:03.325 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:03.325 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:03.325 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:03.325 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:03.325 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:03.325 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:03.326 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:03.326 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:03.326 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:03.326 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:03.326 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:07.325 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:07.326 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:07.327 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:07.328 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:07.328 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:11.329 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:11.330 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:11.330 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:11.330 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:11.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:11.680 7f490d022680 [contrail-kube-manager] [ERROR]: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:45:11.681 7f490d022680 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:45:11.681 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:45:11.681 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=388 09/01/2025 01:45:11.681 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:45:11.681 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 01:45:11.681 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:45:11.682 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'})(timeout=120) 09/01/2025 01:45:11.683 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:45:11.683 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:45:11.683 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:45:11.683 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:45:11.698 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'}) 09/01/2025 01:45:15.331 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:15.332 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:15.333 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:15.333 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:15.333 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:15.333 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:15.333 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:15.333 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:15.333 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:15.334 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:15.334 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:15.334 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:15.334 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:15.334 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:19.334 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:19.336 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:19.336 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:19.336 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:19.336 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:19.336 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:19.336 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:19.336 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:19.336 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:19.337 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:19.337 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:19.337 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:19.337 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:19.337 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:23.336 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:23.337 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:23.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:23.339 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:23.339 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:27.338 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:27.339 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:27.339 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:27.339 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:27.339 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:27.339 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:27.339 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:27.340 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:27.340 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:27.340 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:27.340 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:27.340 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:27.340 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:27.340 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:31.340 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:31.341 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:31.341 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:31.341 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:31.341 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:31.342 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:35.344 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:35.345 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:35.345 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:35.346 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:35.346 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:35.346 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:35.346 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:35.346 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:35.346 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:35.347 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:35.347 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:35.347 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:35.347 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:35.347 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:39.346 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:39.348 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:39.349 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:39.349 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:39.349 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:39.349 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:43.349 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:43.350 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:43.350 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:43.350 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:43.351 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:47.353 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:47.354 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:47.354 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:47.354 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:47.354 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:47.354 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:47.354 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:47.355 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:47.355 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:47.355 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:47.355 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:47.355 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:47.355 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:47.355 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:51.355 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:51.356 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:51.356 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:51.356 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:51.356 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:51.356 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:51.356 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:51.357 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:51.357 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:51.357 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:51.357 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:51.357 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:51.357 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:51.357 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:55.356 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:55.357 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:55.357 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:55.357 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:55.358 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:59.360 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:45:59.361 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:45:59.361 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:45:59.361 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:59.361 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:45:59.361 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:45:59.361 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:45:59.361 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:45:59.361 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:59.362 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:45:59.362 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:59.362 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:59.362 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:45:59.362 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:45:59.909 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:45:59.910 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:45:59.910 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:45:59.910 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:45:59.922 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1013 newResVer=1013 09/01/2025 01:45:59.923 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:45:59.923 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:45:59.923 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:46:00.019 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1042 newResVer=1042 09/01/2025 01:46:00.019 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:46:00.019 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:46:00.019 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:46:01.042 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=385 newResVer=1261 09/01/2025 01:46:01.042 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:46:01.042 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:46:01.042 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:46:03.361 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:03.362 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:03.362 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:03.363 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:07.364 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:07.365 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:07.365 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:07.365 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:07.365 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:07.365 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:07.365 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:07.366 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:07.366 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:07.366 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:07.366 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:07.366 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:07.366 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:07.366 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:11.367 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:11.392 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:11.392 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:11.392 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:11.392 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:11.392 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:11.392 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:11.393 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:11.393 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:11.393 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:11.393 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:11.393 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:11.393 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:11.393 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:15.392 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:15.393 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:15.394 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:15.394 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:15.394 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:15.394 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:15.394 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:15.394 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:15.394 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:15.394 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:15.395 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:15.395 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:15.395 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:15.395 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:19.394 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:19.396 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:19.396 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:19.396 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:19.396 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:19.396 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:19.396 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:19.396 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:19.397 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:19.397 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:19.397 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:19.397 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:19.397 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:19.397 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:23.397 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:23.398 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:23.398 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:23.398 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:23.398 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:23.399 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:27.399 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:27.400 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:27.400 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:27.400 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:27.400 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:27.400 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:27.400 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:27.401 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:27.401 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:27.401 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:27.401 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:27.401 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:27.401 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:27.401 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:31.402 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:31.403 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:31.403 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:31.403 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:31.404 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:35.404 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:35.405 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:35.406 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:35.406 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:35.406 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:35.406 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:35.406 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:35.406 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:35.406 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:35.407 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:35.407 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:35.407 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:35.407 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:35.407 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:39.406 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:39.407 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:39.407 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:39.407 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:39.407 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:39.407 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:39.407 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:39.408 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:39.408 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:39.408 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:39.408 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:39.408 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:39.408 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:39.408 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:43.409 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:43.410 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:43.410 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:43.410 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:43.410 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:43.410 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:43.410 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:43.410 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:43.410 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:43.411 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:43.411 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:43.411 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:43.411 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:43.411 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:43.585 7f490d0cc680 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:46:47.411 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:47.412 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:47.412 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:47.412 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:47.412 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:47.412 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:47.412 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:47.413 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:47.413 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:47.413 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:47.413 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:47.413 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:47.413 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:47.413 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:51.413 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:51.414 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:51.414 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:51.414 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:51.414 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:51.414 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:51.414 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:51.415 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:51.415 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:51.415 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:51.415 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:51.415 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:51.415 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:51.415 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:55.414 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:55.415 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:55.415 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:55.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:59.416 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:46:59.417 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:46:59.417 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:46:59.417 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:46:59.418 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:46:59.948 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1013 newResVer=1013 09/01/2025 01:46:59.948 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:46:59.948 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:46:59.948 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:47:00.002 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:47:00.003 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:47:00.003 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:47:00.003 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:47:00.003 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:47:00.003 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:47:00.003 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:47:00.003 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:47:00.003 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:47:00.011 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:47:00.011 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:47:00.011 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:47:00.012 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:47:00.012 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:47:00.012 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:47:00.012 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:47:00.012 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:47:00.014 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:47:00.014 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:47:00.014 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:47:00.014 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:47:00.014 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:47:00.014 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:47:00.015 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:47:00.015 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:47:00.015 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:47:00.015 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:47:00.015 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:47:00.015 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:47:00.016 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:47:00.017 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:47:00.017 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:47:00.029 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:47:00.030 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:47:00.373 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1042 newResVer=1042 09/01/2025 01:47:00.374 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:47:00.374 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:47:00.374 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:47:00.374 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:47:00.374 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:47:00.375 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:47:00.377 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:47:00.377 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:47:01.636 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=1261 newResVer=1261 09/01/2025 01:47:01.637 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:47:01.637 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:47:01.637 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:47:03.418 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:03.419 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:03.420 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:03.420 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:03.420 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:03.420 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:03.420 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:03.420 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:03.420 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:03.421 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:03.421 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:03.421 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:03.421 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:03.421 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:07.420 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:07.422 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:07.423 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:07.423 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:07.423 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:07.423 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:07.423 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:07.423 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:07.423 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:07.424 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:07.424 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:07.424 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:07.424 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:07.424 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:11.423 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:11.424 7f490d0ed040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:11.424 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:11.424 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:11.424 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:11.425 7f490d0ed040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:11.699 7f490d022680 [contrail-kube-manager] [ERROR]: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:47:11.699 7f490d022680 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:47:11.699 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:47:11.699 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=388 09/01/2025 01:47:11.700 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:47:11.700 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 01:47:11.700 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:47:11.700 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'})(timeout=120) 09/01/2025 01:47:11.701 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:47:11.702 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:47:11.702 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:47:11.702 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:47:11.718 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'}) 09/01/2025 01:47:15.425 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:15.426 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:15.426 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:15.426 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:15.426 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:15.426 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:15.426 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:15.426 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:15.426 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:15.427 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:15.427 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:15.427 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:15.427 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:15.427 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:19.428 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:19.429 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:19.429 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:19.429 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:19.429 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:19.429 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:19.429 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:19.430 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:19.430 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:19.430 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:19.430 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:19.430 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:19.430 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:19.430 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:23.431 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:23.432 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:23.432 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:23.432 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:23.432 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:23.432 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:23.432 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:23.433 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:23.433 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:23.433 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:23.433 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:23.433 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:23.433 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:23.433 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:27.433 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:27.434 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:27.434 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:27.434 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:27.434 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:27.434 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:27.434 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:27.435 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:27.435 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:27.435 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:27.435 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:27.435 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:27.435 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:27.435 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:31.435 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:31.436 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:31.437 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:31.438 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:35.439 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:35.440 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:35.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:35.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:35.440 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:35.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:35.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:35.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:35.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:35.441 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:35.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:35.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:35.441 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:35.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:39.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:39.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:39.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:39.443 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:39.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:43.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:43.444 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:43.444 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:43.444 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:43.444 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:43.444 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:43.444 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:43.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:43.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:43.445 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:43.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:43.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:43.445 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:43.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:47.447 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:47.447 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:47.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:47.449 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:47.449 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:47.449 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:51.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:51.450 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:51.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:51.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:51.450 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:51.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:51.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:51.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:51.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:51.451 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:51.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:51.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:51.451 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:51.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:55.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:55.451 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:55.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:55.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:55.451 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:55.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:55.452 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:55.452 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:55.452 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:55.452 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:55.452 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:55.452 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:55.452 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:55.452 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:59.453 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:47:59.455 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:47:59.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:47:59.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:59.455 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:47:59.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:47:59.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:47:59.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:47:59.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:59.456 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:47:59.456 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:59.456 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:59.456 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:47:59.456 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:47:59.985 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:47:59.985 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:47:59.985 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:47:59.985 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:48:00.318 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1013 newResVer=1013 09/01/2025 01:48:00.318 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:48:00.318 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:48:00.318 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:48:00.318 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:48:00.318 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:48:00.318 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:48:00.319 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:48:00.319 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:48:00.631 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1042 newResVer=1042 09/01/2025 01:48:00.631 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:48:00.631 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:48:00.631 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:48:01.345 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=1261 newResVer=1261 09/01/2025 01:48:01.346 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:48:01.346 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:48:01.346 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:48:03.455 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:03.457 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:03.458 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:03.458 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:07.458 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:07.459 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:07.459 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:07.460 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:10.617 7f490d0ae9a0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:48:11.461 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:11.462 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:11.462 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:11.462 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:11.462 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:11.462 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:11.462 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:11.463 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:11.463 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:11.463 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:11.463 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:11.463 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:11.463 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:11.463 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:13.398 7f490d0ae9a0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:48:15.463 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:15.464 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:15.464 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:15.464 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:15.464 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:15.464 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:15.464 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:15.465 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:15.465 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:15.465 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:15.465 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:15.465 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:15.465 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:15.465 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:16.936 7f490d0ae9a0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:48:19.465 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:19.466 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:19.467 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:19.467 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:19.467 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:19.467 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:19.467 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:19.467 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:19.467 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:19.468 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:19.468 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:19.468 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:19.468 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:19.468 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:23.469 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:23.470 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:23.470 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:23.470 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:23.470 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:23.470 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:23.470 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:23.471 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:23.471 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:23.471 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:23.471 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:23.471 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:23.471 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:23.471 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:27.471 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:27.472 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:27.472 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:27.472 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:27.472 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:27.472 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:27.473 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:27.473 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:27.473 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:27.473 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:27.473 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:27.473 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:27.473 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:27.473 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:31.473 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:31.474 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:31.474 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:31.475 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:35.477 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:35.478 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:35.478 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:35.478 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:35.478 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:35.478 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:35.478 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:35.479 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:35.479 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:35.479 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:35.479 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:35.479 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:35.479 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:35.479 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:39.479 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:39.480 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:39.480 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:39.480 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:39.481 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:43.482 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:43.483 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:43.483 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:43.483 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:43.483 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:43.483 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:43.483 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:43.483 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:43.483 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:43.484 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:43.484 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:43.484 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:43.484 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:43.484 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:43.524 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1042 newResVer=1042 09/01/2025 01:48:43.524 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:48:43.525 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:48:43.525 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:48:44.528 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1042 newResVer=1042 09/01/2025 01:48:44.528 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:48:44.528 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:48:44.528 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:48:45.851 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration NetworkPolicyMonitor 09/01/2025 01:48:45.851 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:48:45.851 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1042 09/01/2025 01:48:45.851 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:48:45.852 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:48:45.852 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:48:45.852 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1042'})(timeout=120) 09/01/2025 01:48:45.853 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:48:45.853 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:48:45.855 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:48:45.856 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:48:45.858 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1042'}) 09/01/2025 01:48:47.484 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:47.485 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:47.486 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:47.487 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:51.486 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:51.487 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:51.487 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:51.488 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:54.193 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1013 newResVer=1013 09/01/2025 01:48:54.194 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:48:54.194 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:48:54.194 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:48:55.488 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:55.489 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:55.489 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:55.489 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:55.489 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:55.489 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:55.489 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:55.489 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:55.490 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:55.490 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:55.490 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:55.490 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:55.490 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:55.490 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:55.873 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration IngressMonitor 09/01/2025 01:48:55.873 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:48:55.873 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1013 09/01/2025 01:48:55.873 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:48:55.873 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 01:48:55.874 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:48:55.874 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1013'})(timeout=120) 09/01/2025 01:48:55.875 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:48:55.875 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:48:55.875 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:48:55.875 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:48:55.880 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1013'}) 09/01/2025 01:48:59.466 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:48:59.466 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:48:59.466 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:48:59.466 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:48:59.490 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:48:59.491 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:59.492 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:48:59.492 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:48:59.492 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:00.031 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:49:00.031 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:49:00.031 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:49:00.032 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:49:00.032 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:49:00.032 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:49:00.032 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:49:00.032 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:49:00.032 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:49:00.032 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:49:00.033 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:49:00.033 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:49:00.033 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:49:00.034 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:49:00.034 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:49:00.034 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:49:00.035 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:49:00.035 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:49:00.035 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:49:00.036 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:49:00.036 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:49:00.036 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:49:00.036 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:49:00.047 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:49:00.047 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:49:01.845 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=1261 newResVer=1261 09/01/2025 01:49:01.845 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:49:01.845 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:49:01.845 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:49:01.845 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:49:01.846 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:49:01.846 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:49:01.846 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:49:03.492 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:03.493 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:03.493 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:03.493 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:03.493 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:03.493 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:03.493 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:03.494 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:03.494 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:03.494 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:03.494 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:03.494 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:03.494 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:03.494 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:07.494 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:07.495 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:07.495 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:07.496 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:11.497 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:11.498 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:11.498 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:11.498 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:11.499 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:11.719 7f490d022680 [contrail-kube-manager] [ERROR]: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:49:11.720 7f490d022680 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:49:11.720 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:49:11.721 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=388 09/01/2025 01:49:11.721 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:49:11.721 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 01:49:11.721 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:49:11.721 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'})(timeout=120) 09/01/2025 01:49:11.722 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:49:11.723 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:49:11.723 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:49:11.723 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:49:11.740 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'}) 09/01/2025 01:49:15.499 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:15.500 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:15.500 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:15.500 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:15.500 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:15.500 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:15.500 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:15.501 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:15.501 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:15.501 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:15.501 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:15.501 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:15.501 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:15.501 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:19.500 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:19.502 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:19.502 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:19.502 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:19.502 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:19.502 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:19.502 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:19.502 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:19.503 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:19.503 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:19.503 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:19.503 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:19.503 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:19.503 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:23.503 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:23.504 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:23.504 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:23.504 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:23.504 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:23.504 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:23.504 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:23.505 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:23.505 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:23.505 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:23.505 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:23.505 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:23.505 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:23.505 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:27.505 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:27.506 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:27.506 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:27.506 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:27.507 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:31.507 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:31.508 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:31.508 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:31.508 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:31.508 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:31.508 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:31.508 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:31.509 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:31.509 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:31.509 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:31.509 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:31.509 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:31.509 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:31.509 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:35.510 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:35.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:35.512 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:35.512 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:39.511 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:39.512 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:39.512 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:39.512 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:39.512 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:39.512 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:39.512 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:39.513 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:39.513 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:39.513 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:39.513 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:39.513 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:39.513 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:39.513 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:43.514 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:43.515 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:43.515 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:43.515 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:43.515 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:43.516 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:47.518 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:47.519 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:47.519 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:47.519 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:47.519 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:47.519 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:47.519 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:47.520 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:47.520 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:47.520 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:47.520 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:47.520 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:47.520 7f490d0cc7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:47.520 7f490d0cc7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:51.519 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:51.520 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:51.521 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:55.522 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:55.523 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:55.524 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:55.524 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:55.524 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:59.132 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:49:59.133 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:49:59.133 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:49:59.133 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:49:59.523 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:49:59.524 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:49:59.525 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:01.530 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=1261 newResVer=1261 09/01/2025 01:50:01.530 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:50:01.530 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:50:01.530 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:50:03.525 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:03.526 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:03.527 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:03.528 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:03.528 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:07.528 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:07.529 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:07.529 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:07.529 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:07.529 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:07.529 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:07.529 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:07.530 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:07.530 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:07.530 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:07.530 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:07.530 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:07.530 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:07.530 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:11.530 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:11.531 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:11.532 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:11.533 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:15.533 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:15.534 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:15.534 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:15.534 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:15.534 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:15.534 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:15.534 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:15.534 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:15.534 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:15.535 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:15.535 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:15.535 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:15.535 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:15.535 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:19.536 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:19.537 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:19.538 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:19.538 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:23.538 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:23.539 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:23.540 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:27.541 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:27.542 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:27.543 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:27.543 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:27.543 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:27.543 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:31.543 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:31.544 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:31.544 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:31.544 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:31.544 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:31.544 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:31.544 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:31.545 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:31.545 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:31.545 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:31.545 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:31.545 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:31.545 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:31.545 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:35.547 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:35.547 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:35.548 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:39.549 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:39.550 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:39.550 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:39.550 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:39.550 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:39.550 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:39.550 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:39.550 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:39.551 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:39.551 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:39.551 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:39.551 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:39.551 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:39.551 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:43.550 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:43.551 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:43.551 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:43.552 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:45.859 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:50:45.860 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:50:45.860 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:50:45.860 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1042 09/01/2025 01:50:45.860 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:50:45.860 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:50:45.860 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:50:45.860 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1042'})(timeout=120) 09/01/2025 01:50:45.862 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:50:45.862 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:50:45.864 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:50:45.864 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:50:45.868 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1042'}) 09/01/2025 01:50:47.553 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:47.554 7f490d0ae860 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:47.554 7f490d0ae860 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:47.554 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:47.554 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:47.554 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:47.554 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:47.554 7f490d0ae860 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:47.554 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:47.555 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:47.555 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:47.555 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:47.555 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:47.555 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:51.554 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:51.556 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:51.557 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:51.557 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:55.557 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:55.558 7f490d0ae860 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:55.558 7f490d0ae860 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:55.558 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:55.558 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:55.558 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:55.558 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:55.558 7f490d0ae860 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:55.558 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:55.559 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:55.559 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:55.559 7f490d0ae860 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:55.559 7f490d0ae860 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:55.559 7f490d0ae860 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:55.881 7f490d079ae0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:50:55.881 7f490d079ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:50:55.881 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:50:55.881 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1013 09/01/2025 01:50:55.882 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:50:55.882 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 01:50:55.882 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:50:55.882 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1013'})(timeout=120) 09/01/2025 01:50:55.883 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:50:55.883 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:50:55.883 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:50:55.883 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:50:55.889 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1013'}) 09/01/2025 01:50:59.561 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:50:59.562 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:59.563 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:59.563 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:50:59.563 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:50:59.831 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:50:59.831 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:50:59.831 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:50:59.831 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:51:00.048 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:51:00.048 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:51:00.049 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:51:00.049 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:51:00.049 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:51:00.049 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:51:00.049 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:51:00.049 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:51:00.050 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:51:00.050 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:51:00.050 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:51:00.050 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:51:00.050 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:51:00.051 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:51:00.051 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:51:00.051 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:51:00.052 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:51:00.052 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:51:00.053 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:51:00.053 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:51:00.053 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:51:00.053 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:51:00.053 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:51:00.063 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:51:00.064 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:51:01.068 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=1261 newResVer=1261 09/01/2025 01:51:01.068 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:51:01.068 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:51:01.068 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:51:03.563 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:03.564 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:03.564 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:03.564 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:03.564 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:03.564 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:03.564 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:03.565 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:03.565 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:03.565 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:03.565 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:03.565 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:03.565 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:03.565 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:07.565 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:07.566 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:07.567 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:11.568 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:11.569 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:11.570 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:11.571 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:11.571 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:11.571 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:11.741 7f490d022680 [contrail-kube-manager] [ERROR]: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:51:11.742 7f490d022680 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:51:11.742 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:51:11.742 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=388 09/01/2025 01:51:11.742 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:51:11.743 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 01:51:11.743 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:51:11.743 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'})(timeout=120) 09/01/2025 01:51:11.744 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:51:11.745 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:51:11.745 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:51:11.745 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:51:11.757 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'}) 09/01/2025 01:51:15.571 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:15.572 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:15.572 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:15.572 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:15.572 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:15.572 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:15.572 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:15.572 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:15.572 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:15.573 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:15.573 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:15.573 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:15.573 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:15.573 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:19.572 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:19.573 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:19.573 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:19.574 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:23.576 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:23.577 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:23.577 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:23.577 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:23.577 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:23.577 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:23.577 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:23.578 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:23.578 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:23.578 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:23.578 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:23.578 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:23.578 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:23.578 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:27.578 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:27.579 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:27.579 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:27.579 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:27.579 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:27.579 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:27.579 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:27.580 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:27.580 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:27.580 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:27.580 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:27.580 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:27.580 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:27.580 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:31.581 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:31.582 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:31.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:35.583 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:35.585 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:35.586 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:35.586 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:35.586 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:39.586 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:39.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:39.588 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:39.588 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:43.589 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:43.590 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:43.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:43.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:43.590 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:43.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:43.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:43.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:43.591 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:43.591 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:43.591 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:43.591 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:43.591 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:43.591 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:47.592 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:47.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:47.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:47.594 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:47.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:50.064 7f490d0ed180 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:51:51.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:51.594 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:51.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:51.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:51.594 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:51.595 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:55.595 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:55.596 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:55.597 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:59.402 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:51:59.403 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:51:59.403 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:51:59.403 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:51:59.403 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:51:59.403 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:51:59.403 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:51:59.403 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:51:59.403 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:51:59.597 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:51:59.598 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:51:59.598 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:51:59.598 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:59.598 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:51:59.598 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:51:59.598 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:51:59.599 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:51:59.599 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:59.599 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:51:59.599 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:59.599 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:51:59.599 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:51:59.599 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:01.237 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=1261 newResVer=1261 09/01/2025 01:52:01.237 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:52:01.237 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:52:01.237 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:52:03.599 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:03.600 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:03.600 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:03.600 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:03.600 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:03.600 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:03.600 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:03.601 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:03.601 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:03.601 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:03.601 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:03.601 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:03.601 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:03.601 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:07.602 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:07.603 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:07.603 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:07.603 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:07.604 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:11.606 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:11.607 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:11.607 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:11.607 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:11.607 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:11.607 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:11.607 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:11.608 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:11.608 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:11.608 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:11.608 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:11.608 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:11.608 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:11.608 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:12.158 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:52:12.158 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:52:12.159 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:52:12.159 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:52:13.218 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1052 09/01/2025 01:52:13.218 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:52:13.219 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:52:13.219 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:52:14.813 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration ServiceMonitor 09/01/2025 01:52:14.814 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:52:14.814 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1052 09/01/2025 01:52:14.814 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:52:14.814 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 01:52:14.814 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:52:14.815 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1052'})(timeout=120) 09/01/2025 01:52:14.816 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:52:14.816 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:52:14.816 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:52:14.816 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:52:14.830 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1052'}) 09/01/2025 01:52:15.608 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:15.609 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:15.610 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:15.610 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:15.610 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:15.610 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:15.610 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:15.610 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:15.610 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:15.610 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:15.611 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:15.611 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:15.611 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:15.611 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:19.610 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:19.611 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:19.612 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:23.614 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:23.616 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:23.617 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:23.617 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:27.617 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:27.618 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:27.618 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:27.618 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:27.618 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:27.618 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:27.618 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:27.619 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:27.619 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:27.619 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:27.619 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:27.619 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:27.619 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:27.619 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:31.620 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:31.621 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:31.622 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:31.622 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:31.622 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:35.624 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:35.625 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:35.625 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:35.625 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:35.625 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:35.625 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:35.625 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:35.626 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:35.626 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:35.626 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:35.626 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:35.626 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:35.626 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:35.626 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:39.626 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:39.627 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:39.627 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:39.627 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:39.627 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:39.628 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:43.628 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:43.629 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:43.630 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:43.631 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:43.631 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:45.870 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:52:45.870 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:52:45.870 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:52:45.870 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1042 09/01/2025 01:52:45.871 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:52:45.871 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:52:45.871 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:52:45.871 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1042'})(timeout=120) 09/01/2025 01:52:45.872 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:52:45.872 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:52:45.875 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:52:45.875 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:52:45.879 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1042'}) 09/01/2025 01:52:47.630 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:47.631 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:47.631 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:47.631 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:47.632 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:51.632 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:51.633 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:51.633 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:51.633 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:51.633 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:51.633 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:51.634 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:51.634 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:51.634 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:51.634 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:51.634 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:51.634 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:51.634 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:51.634 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:55.634 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:55.635 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:55.635 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:55.635 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:55.636 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:55.891 7f490d079ae0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:52:55.891 7f490d079ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:52:55.891 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:52:55.892 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1013 09/01/2025 01:52:55.892 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:52:55.892 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 01:52:55.892 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:52:55.892 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1013'})(timeout=120) 09/01/2025 01:52:55.893 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:52:55.893 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:52:55.894 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:52:55.894 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:52:55.900 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1013'}) 09/01/2025 01:52:56.857 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=1261 newResVer=1261 09/01/2025 01:52:56.858 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:52:56.858 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:52:56.858 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:52:58.720 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration NetworkMonitor 09/01/2025 01:52:58.720 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:52:58.720 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1261 09/01/2025 01:52:58.720 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:52:58.720 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:52:58.721 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:52:58.721 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1261'})(timeout=120) 09/01/2025 01:52:58.722 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:52:58.722 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:52:58.722 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:52:58.727 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1261'}) 09/01/2025 01:52:59.637 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:59.638 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:52:59.639 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:59.639 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:52:59.639 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:52:59.639 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:00.065 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:53:00.065 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:53:00.065 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:53:00.065 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:53:00.065 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:53:00.066 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:53:00.066 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:53:00.066 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:53:00.066 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=381 09/01/2025 01:53:00.066 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:53:00.066 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 01:53:00.066 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:53:00.067 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:53:00.068 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:53:00.068 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:53:00.068 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'})(timeout=120) 09/01/2025 01:53:00.069 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:53:00.069 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:53:00.069 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=1 timeout=120) 09/01/2025 01:53:00.069 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:53:00.069 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:53:00.069 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:53:00.069 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:53:00.076 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:53:00.077 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '381'}) 09/01/2025 01:53:03.639 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:03.640 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:03.640 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:03.640 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:03.640 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:03.640 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:03.640 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:03.641 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:03.641 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:03.641 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:03.641 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:03.641 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:03.641 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:03.641 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:07.642 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:07.644 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:07.644 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:07.644 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:07.644 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:07.644 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:07.644 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:07.645 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:07.645 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:07.645 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:07.645 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:07.645 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:07.645 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:07.645 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:11.646 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:11.648 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:11.648 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:11.648 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:11.648 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:11.648 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:11.648 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:11.648 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:11.649 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:11.649 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:11.649 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:11.649 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:11.649 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:11.649 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:11.758 7f490d022680 [contrail-kube-manager] [ERROR]: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:53:11.758 7f490d022680 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:53:11.758 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:53:11.758 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=388 09/01/2025 01:53:11.759 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:53:11.759 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 01:53:11.759 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:53:11.759 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'})(timeout=120) 09/01/2025 01:53:11.761 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:53:11.761 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:53:11.761 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:53:11.761 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:53:11.774 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '388'}) 09/01/2025 01:53:15.649 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:15.650 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:15.650 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:15.650 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:15.650 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:15.650 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:15.651 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:15.651 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:15.651 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:15.651 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:15.651 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:15.651 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:15.651 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:15.651 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:19.650 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:19.651 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:19.651 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:19.651 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:19.651 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:19.652 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:23.652 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:23.653 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:23.654 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:23.654 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:23.654 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:23.654 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:23.654 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:23.654 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:23.654 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:23.655 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:23.655 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:23.655 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:23.655 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:23.655 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:27.654 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:27.655 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:27.655 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:27.655 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:27.655 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:27.655 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:27.656 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:27.656 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:27.656 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:27.656 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:27.656 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:27.656 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:27.656 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:27.656 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:31.657 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:31.658 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:31.658 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:31.658 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:31.658 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:31.658 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:31.658 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:31.659 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:31.659 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:31.659 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:31.659 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:31.659 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:31.659 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:31.659 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:35.661 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:35.662 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:35.662 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:35.662 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:35.663 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:39.663 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:39.664 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:39.664 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:39.664 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:39.664 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:39.664 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:39.664 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:39.665 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:39.665 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:39.665 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:39.665 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:39.665 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:39.665 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:39.665 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:43.665 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:43.666 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:43.667 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:43.668 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:43.668 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:45.648 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1042 newResVer=1932 09/01/2025 01:53:45.649 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:53:45.649 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:53:45.649 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:53:47.668 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:47.669 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:47.669 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:47.669 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:47.669 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:47.669 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:47.669 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:47.669 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:47.669 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:47.670 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:47.670 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:47.670 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:47.670 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:47.670 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:49.532 7f490d0ae040 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:53:51.670 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:51.672 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:51.672 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:51.672 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:51.672 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:51.672 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:51.672 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:51.672 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:51.673 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:51.673 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:51.673 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:51.673 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:51.673 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:51.673 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:51.920 7f490d0ae040 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:53:55.461 7f490d0ae040 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:53:55.673 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:55.674 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:55.674 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:55.674 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:55.674 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:55.674 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:55.675 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:55.675 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:55.675 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:55.675 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:55.675 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:55.675 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:55.675 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:55.675 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:56.236 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1013 newResVer=1872 09/01/2025 01:53:56.237 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:53:56.237 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:53:56.237 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:53:59.675 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:53:59.676 7f490d022b80 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:53:59.676 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:53:59.676 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:59.676 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:53:59.676 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:53:59.676 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:53:59.677 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:53:59.677 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:59.677 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:53:59.677 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:59.677 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:53:59.677 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:53:59.677 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:00.277 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=381 newResVer=1948 09/01/2025 01:54:00.278 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:54:00.278 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:54:00.278 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:54:00.289 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=381 newResVer=1946 09/01/2025 01:54:00.289 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:54:00.289 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:54:00.289 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:54:03.677 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:03.678 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:03.678 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:03.678 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:03.678 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:03.678 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:03.678 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:03.679 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:03.679 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:03.679 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:03.679 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:03.679 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:03.679 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:03.679 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:07.680 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:07.681 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:07.681 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:07.681 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:07.682 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:07.682 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:07.682 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:07.682 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:07.682 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:07.682 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:07.682 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:07.682 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:07.683 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:07.683 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:11.682 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:11.684 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:11.684 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:11.684 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:11.684 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:11.684 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:11.684 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:11.684 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:11.685 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:11.685 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:11.685 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:11.685 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:11.685 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:11.685 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:12.173 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=388 newResVer=1983 09/01/2025 01:54:12.174 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:54:12.174 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:54:12.174 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:54:14.833 7f490d0224a0 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:54:14.833 7f490d0224a0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:54:14.834 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:54:14.834 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1052 09/01/2025 01:54:14.834 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:54:14.834 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 01:54:14.834 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:54:14.834 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1052'})(timeout=120) 09/01/2025 01:54:14.836 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:54:14.836 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:54:14.836 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:54:14.836 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:54:14.859 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1052'}) 09/01/2025 01:54:15.684 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:15.685 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:15.685 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:15.685 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:15.686 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:19.686 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:19.687 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:19.687 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:19.688 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:23.690 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:23.691 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:23.691 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:23.691 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:23.692 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:27.692 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:27.693 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:27.693 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:27.693 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:27.694 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:31.695 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:31.696 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:31.696 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:31.696 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:31.696 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:31.696 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:31.696 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:31.697 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:31.697 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:31.697 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:31.697 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:31.697 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:31.697 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:31.697 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:35.699 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:35.700 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:35.701 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:35.702 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:39.702 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:39.703 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:39.703 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:39.703 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:39.703 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:39.703 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:39.703 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:39.703 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:39.703 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:39.704 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:39.704 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:39.704 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:39.704 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:39.704 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:43.705 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:43.706 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:43.706 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:43.706 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:43.706 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:43.706 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:43.706 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:43.707 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:43.707 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:43.707 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:43.707 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:43.707 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:43.707 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:43.707 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:45.770 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1932 newResVer=1932 09/01/2025 01:54:45.770 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:54:45.771 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:54:45.771 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:54:47.708 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:47.709 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:47.709 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:47.709 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:47.709 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:47.709 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:47.709 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:47.710 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:47.710 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:47.710 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:47.710 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:47.710 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:47.710 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:47.710 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:51.710 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:51.711 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:51.711 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:51.711 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:51.711 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:51.711 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:51.711 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:51.711 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:51.711 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:51.712 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:51.712 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:51.712 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:51.712 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:51.712 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:55.406 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 01:54:55.406 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:54:55.406 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:54:55.406 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:54:55.712 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:55.713 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:55.713 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:55.713 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:55.713 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:55.713 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:55.713 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:55.714 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:55.714 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:55.714 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:55.714 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:55.714 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:55.714 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:55.714 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:58.730 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:54:58.730 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 01:54:58.730 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:54:58.731 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1261 09/01/2025 01:54:58.731 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:54:58.731 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 01:54:58.731 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:54:58.731 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1261'})(timeout=120) 09/01/2025 01:54:58.732 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:54:58.733 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:54:58.733 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:54:58.742 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1261'}) 09/01/2025 01:54:59.714 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:54:59.715 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:54:59.716 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:00.321 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:55:00.321 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:55:00.321 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:55:00.321 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:55:00.341 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=1946 newResVer=1946 09/01/2025 01:55:00.341 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:55:00.341 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:55:00.341 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:55:03.717 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:03.718 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:03.718 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:03.718 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:03.718 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:03.718 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:03.718 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:03.718 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:03.718 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:03.719 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:03.719 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:03.719 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:03.719 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:03.719 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:07.719 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:07.720 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:07.720 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:07.721 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:11.723 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:11.724 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:11.724 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:11.724 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:11.725 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:12.787 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=1983 09/01/2025 01:55:12.787 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:55:12.787 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:55:12.787 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:55:14.469 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1052 newResVer=1948 09/01/2025 01:55:14.469 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:55:14.469 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:55:14.469 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:55:15.725 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:15.726 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:15.726 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:15.726 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:15.726 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:15.726 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:15.726 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:15.727 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:15.727 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:15.727 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:15.727 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:15.727 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:15.727 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:15.727 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:19.727 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:19.728 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:19.728 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:19.728 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:19.728 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:19.729 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:23.731 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:23.732 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:23.733 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:23.733 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:23.733 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:23.733 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:27.733 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:27.734 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:27.734 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:27.734 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:27.734 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:27.734 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:27.734 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:27.735 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:27.735 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:27.735 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:27.735 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:27.735 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:27.735 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:27.735 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:31.735 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:31.736 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:31.736 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:31.736 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:31.736 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:31.736 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:31.736 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:31.737 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:31.737 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:31.737 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:31.737 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:31.737 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:31.737 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:31.737 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:35.738 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:35.739 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:35.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:39.740 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:39.741 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:39.741 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:39.741 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:39.742 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:43.743 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:43.744 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:43.744 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:43.744 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:43.744 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:43.744 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:43.744 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:43.745 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:43.745 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:43.745 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:43.745 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:43.745 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:43.745 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:43.745 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:45.716 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1932 newResVer=1932 09/01/2025 01:55:45.716 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:55:45.716 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:55:45.716 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:55:47.746 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:47.747 7f490d022b80 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:47.747 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:47.747 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:47.747 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:47.747 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:47.747 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:47.747 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:47.747 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:47.748 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:47.748 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:47.748 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:47.748 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:47.748 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:51.747 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:51.748 7f490d022b80 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:51.748 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:51.748 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:51.748 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:51.748 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:51.749 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:51.749 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:51.749 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:51.749 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:51.749 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:51.749 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:51.749 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:51.749 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:55.749 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:55.750 7f490d022b80 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:55.750 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:55.750 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:55.750 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:55.750 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:55.750 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:55.751 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:55.751 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:55.751 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:55.751 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:55.751 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:55.751 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:55.751 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:56.186 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 01:55:56.186 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:55:56.186 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:55:56.186 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:55:57.997 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=1261 newResVer=2103 09/01/2025 01:55:57.998 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:55:57.998 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:55:57.998 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:55:59.750 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:55:59.751 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:55:59.751 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:55:59.752 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:00.944 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=1946 newResVer=1946 09/01/2025 01:56:00.944 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:56:00.944 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:56:00.944 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:56:01.062 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:56:01.062 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:56:01.062 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:56:01.062 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:56:03.752 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:03.754 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:03.754 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:03.754 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:03.754 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:03.754 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:03.754 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:03.755 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:03.755 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:03.755 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:03.755 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:03.755 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:03.755 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:03.755 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:07.756 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:07.757 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:07.757 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:07.757 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:07.758 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:11.758 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:11.759 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:11.760 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:12.775 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=1983 09/01/2025 01:56:12.775 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:56:12.775 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:56:12.775 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:56:14.755 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:56:14.755 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:56:14.755 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:56:14.755 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:56:15.761 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.762 7f490d10ce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:15.763 7f490d10ce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.763 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:15.763 7f490d10ce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:15.763 7f490d10ce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:19.762 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:19.763 7f490d10ce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:19.763 7f490d10ce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:19.764 7f490d10ce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:23.764 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:23.765 7f490d10c5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:23.765 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:23.766 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:23.767 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:23.767 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:27.766 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:27.767 7f490d10c5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:27.767 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:27.767 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:27.767 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:27.767 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:27.767 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:27.768 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:27.768 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:27.768 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:27.768 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:27.768 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:27.768 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:27.768 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:28.621 7f490d10c5e0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:56:31.769 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:31.770 7f490d10c5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:31.770 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:31.770 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:31.770 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:31.770 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:31.770 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:31.771 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:31.771 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:31.771 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:31.771 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:31.771 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:31.771 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:31.771 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:35.771 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:35.772 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:35.773 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:35.773 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:39.773 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:39.774 7f490d10c5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:39.774 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:39.774 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:39.774 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:39.774 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:39.774 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:39.775 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:39.775 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:39.775 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:39.775 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:39.775 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:39.775 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:39.775 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:43.775 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:43.776 7f490d10c5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:43.776 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:43.776 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:43.776 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:43.776 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:43.776 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:43.777 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:43.777 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:43.777 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:43.777 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:43.777 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:43.777 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:43.777 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:45.356 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1932 newResVer=1932 09/01/2025 01:56:45.357 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:56:45.357 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:56:45.357 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:56:47.779 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:47.780 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:47.780 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:47.780 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:47.781 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:51.782 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:51.783 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:51.783 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:51.783 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:51.783 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:51.783 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:51.783 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:51.783 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:51.783 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:51.784 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:51.784 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:51.784 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:51.784 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:51.784 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.723 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 01:56:55.723 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:56:55.724 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:56:55.724 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:56:55.783 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:55.784 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:55.784 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:55.784 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:55.784 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:55.784 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:55.784 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:55.785 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:55.785 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.785 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:55.785 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.785 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:55.785 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:55.785 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:58.401 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2103 09/01/2025 01:56:58.401 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:56:58.401 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:56:58.401 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:56:58.734 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:56:59.785 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:56:59.786 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:56:59.786 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:56:59.786 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:59.786 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:56:59.786 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:56:59.786 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:56:59.787 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:56:59.787 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:59.787 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:56:59.787 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:59.787 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:56:59.787 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:56:59.787 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:00.902 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=1946 newResVer=1946 09/01/2025 01:57:00.902 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:57:00.902 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:57:00.902 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:57:00.902 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:57:00.902 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:57:00.902 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:57:00.902 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:57:00.903 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:57:01.666 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:57:01.666 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:57:01.666 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 01:57:01.666 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:57:01.666 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 01:57:01.667 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 01:57:01.667 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:57:01.667 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:57:03.787 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:03.788 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:03.788 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:03.788 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:03.788 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:03.788 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:03.788 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:03.788 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:03.789 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:03.789 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:03.789 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:03.789 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:03.789 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:03.789 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.790 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:07.791 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:07.791 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:07.791 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:07.791 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:07.791 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:07.791 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:07.792 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:07.792 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.792 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:07.792 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.792 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:07.792 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:07.792 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:11.794 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:11.795 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:11.795 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:11.795 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:11.796 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:12.937 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=1983 09/01/2025 01:57:12.938 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:57:12.938 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 01:57:12.938 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:57:12.938 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 01:57:12.938 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 01:57:12.938 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 01:57:12.938 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:57:12.938 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:57:14.641 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:57:14.644 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:57:14.644 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:57:14.644 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:57:15.796 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:15.797 7f490d13ff40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:15.797 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:15.797 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:15.797 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:15.797 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:15.797 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:15.798 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:15.798 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.798 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:15.798 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.798 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:15.798 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:15.798 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.798 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:19.799 7f490d13ff40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:19.799 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:19.800 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.803 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:23.805 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:23.805 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.804 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:27.805 7f490d13ff40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:27.805 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:27.805 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:27.805 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:27.805 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:27.805 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:27.806 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:27.806 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.806 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:27.806 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.806 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:27.806 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:27.806 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.807 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.809 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:31.810 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.810 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:31.810 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:31.810 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.811 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:35.812 7f490d13ff40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:35.812 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:35.812 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:35.813 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.813 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:39.814 7f490d13ff40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:39.814 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:39.814 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:39.814 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:39.814 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:39.814 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:39.815 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:39.815 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.815 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:39.815 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.815 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:39.815 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:39.815 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.816 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:43.817 7f490d13ff40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:43.817 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:43.817 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:43.818 7f490d13ff40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:45.337 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1932 newResVer=1932 09/01/2025 01:57:45.337 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:57:45.337 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:57:45.338 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:57:45.338 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:57:45.338 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:57:45.338 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:57:45.340 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:57:45.340 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:57:47.819 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:47.820 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:47.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:51.821 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:51.822 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:51.823 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:51.823 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:51.823 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:51.823 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:51.823 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:51.823 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:51.823 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:51.824 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:51.824 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:51.824 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:51.824 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:51.824 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.469 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 01:57:55.469 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:57:55.469 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 01:57:55.470 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:57:55.470 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 01:57:55.470 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 01:57:55.470 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 01:57:55.470 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:57:55.470 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:57:55.824 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:55.825 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.826 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:55.826 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:55.826 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:58.647 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2103 09/01/2025 01:57:58.647 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:57:58.647 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:57:58.647 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:57:59.826 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:57:59.827 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:57:59.827 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:57:59.828 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:00.483 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=1946 newResVer=1946 09/01/2025 01:58:00.483 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:58:00.483 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:58:00.483 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:58:02.079 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:58:02.080 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:58:02.080 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:58:02.080 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:58:03.828 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:03.829 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:03.829 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:03.829 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:03.830 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.831 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:07.832 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:07.832 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:07.832 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:07.832 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:07.832 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:07.832 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:07.833 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:07.833 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.833 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:07.833 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.833 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:07.833 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:07.833 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.834 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:11.835 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:11.836 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:12.709 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=1983 09/01/2025 01:58:12.709 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:58:12.709 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:58:12.709 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:58:14.721 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:58:14.721 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:58:14.721 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:58:14.721 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:58:15.836 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:15.837 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:15.837 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:15.837 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:15.837 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:15.837 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:15.838 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:15.838 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:15.838 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.838 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:15.838 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.838 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:15.838 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:15.838 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.839 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:19.840 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:19.840 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:19.840 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:19.840 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:19.840 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:19.840 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:19.840 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:19.840 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.841 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:19.841 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.841 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:19.841 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:19.841 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.579 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=1946 newResVer=1946 09/01/2025 01:58:23.580 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:58:23.580 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:58:23.580 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:58:23.841 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:23.842 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.843 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:23.843 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:23.843 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:25.252 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration NamespaceMonitor 09/01/2025 01:58:25.253 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 01:58:25.253 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1946 09/01/2025 01:58:25.253 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:58:25.253 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 01:58:25.253 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 01:58:25.253 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1946'})(timeout=120) 09/01/2025 01:58:25.255 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 01:58:25.255 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 01:58:25.255 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 01:58:25.255 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:58:25.274 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1946'}) 09/01/2025 01:58:27.842 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:27.843 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:27.844 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.845 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:31.846 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:31.846 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:31.847 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.848 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:35.849 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:35.849 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:35.849 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:35.849 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:35.849 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:35.849 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:35.849 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:35.849 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.850 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:35.850 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.850 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:35.850 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:35.850 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:37.577 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1932 newResVer=1932 09/01/2025 01:58:37.577 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:58:37.577 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:58:37.577 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:58:39.438 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration NetworkPolicyMonitor 09/01/2025 01:58:39.439 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 01:58:39.439 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1932 09/01/2025 01:58:39.439 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 01:58:39.439 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 01:58:39.439 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 01:58:39.439 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1932'})(timeout=120) 09/01/2025 01:58:39.440 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 01:58:39.440 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 01:58:39.442 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 01:58:39.443 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:58:39.446 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1932'}) 09/01/2025 01:58:39.849 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:39.850 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.851 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:39.852 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:39.852 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.852 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:43.853 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:43.853 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:43.853 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:43.853 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:43.853 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:43.853 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:43.854 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:43.854 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.854 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:43.854 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.854 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:43.854 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:43.854 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.855 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:47.856 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:47.856 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:47.856 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:47.857 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.857 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:51.858 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:51.858 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:51.858 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:51.858 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:51.859 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.298 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 01:58:55.298 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:58:55.299 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:58:55.299 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:58:55.860 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:55.860 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:55.861 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:57.944 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2103 09/01/2025 01:58:57.944 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:58:57.945 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:58:57.945 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:58:59.862 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:58:59.863 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:58:59.863 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:58:59.863 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:59.863 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:58:59.863 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:58:59.863 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:58:59.864 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:58:59.864 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.864 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:58:59.864 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.864 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:58:59.864 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:58:59.864 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.236 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:59:03.236 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:59:03.237 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:59:03.237 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 01:59:03.864 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:03.865 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:03.865 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:03.865 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:03.865 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:03.865 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:03.865 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:03.865 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:03.865 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.866 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:03.866 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.866 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:03.866 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:03.866 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.866 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:07.867 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:07.868 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:07.869 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.869 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:11.870 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:11.871 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:11.872 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:12.851 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=1983 09/01/2025 01:59:12.851 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:59:12.852 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:59:12.852 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 01:59:14.249 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 01:59:14.250 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:59:14.250 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 01:59:14.250 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:59:14.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 01:59:14.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 01:59:14.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 01:59:14.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:59:14.250 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 01:59:15.872 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:15.873 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.874 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:15.874 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:15.874 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.874 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:19.875 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:19.875 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:19.875 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:19.876 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.876 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:23.877 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:23.877 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:23.877 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:23.877 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:23.877 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:23.877 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:23.878 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:23.878 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.878 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:23.878 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.878 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:23.878 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:23.878 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.878 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:27.879 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:27.879 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:27.879 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:27.879 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:27.879 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:27.879 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:27.880 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:27.880 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.880 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:27.880 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.880 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:27.880 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:27.880 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:30.849 7f490d1552c0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:59:31.880 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:31.882 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:31.882 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:31.882 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:31.882 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:31.882 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:31.882 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:31.883 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:31.883 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.883 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:31.883 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.883 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:31.883 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:31.883 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:33.179 7f490d1552c0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:59:35.884 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:35.885 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:35.885 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:35.885 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:35.885 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:35.885 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:35.885 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:35.886 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:35.886 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.886 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:35.886 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.886 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:35.886 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:35.886 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:36.753 7f490d1552c0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 01:59:39.886 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.887 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:39.888 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.888 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:39.888 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:39.888 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.889 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:43.891 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:43.891 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:43.891 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:43.891 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:43.891 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:43.891 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:43.892 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:43.892 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.892 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:43.892 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.892 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:43.892 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:43.892 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.892 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:47.893 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:47.893 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:47.893 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:47.893 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:47.893 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:47.893 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:47.893 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:47.893 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.894 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:47.894 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.894 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:47.894 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:47.894 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.894 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:51.895 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:51.895 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:51.895 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:51.895 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:51.895 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:51.895 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:51.896 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:51.896 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.896 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:51.896 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.896 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:51.896 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:51.896 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.824 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 01:59:55.825 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:59:55.825 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:59:55.825 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 01:59:55.895 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:55.896 7f490d1557c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:55.896 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:55.896 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:55.897 7f490d1557c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:58.775 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2103 09/01/2025 01:59:58.775 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:59:58.775 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 01:59:58.776 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 01:59:58.776 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 01:59:58.776 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 01:59:58.776 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 01:59:58.776 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 01:59:59.898 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 01:59:59.899 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 01:59:59.899 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 01:59:59.899 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 01:59:59.900 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.900 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:03.901 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.902 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.902 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:03.902 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:03.963 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:00:03.963 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:00:03.963 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:00:03.963 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:00:07.902 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:07.903 7f490e2bae00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:07.903 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:07.903 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:07.904 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:07.904 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:07.904 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:07.904 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:07.904 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:07.904 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:07.904 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:07.905 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:07.905 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:07.905 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.906 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:11.907 7f490e2bae00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:11.907 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:11.907 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:11.907 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:11.907 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:11.908 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:11.908 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:11.908 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.908 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:11.908 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.908 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:11.908 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:11.908 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:13.105 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=1983 09/01/2025 02:00:13.106 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:00:13.106 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:00:13.106 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:00:14.928 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:00:14.928 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:00:14.928 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:00:14.928 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:00:15.908 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:15.909 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:15.910 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:15.910 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:15.910 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.910 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:19.911 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:19.911 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:19.912 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.914 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.915 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:23.916 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:23.916 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:25.276 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:00:25.276 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:00:25.276 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:00:25.276 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1946 09/01/2025 02:00:25.277 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:00:25.277 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 02:00:25.277 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 02:00:25.277 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1946'})(timeout=120) 09/01/2025 02:00:25.278 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 02:00:25.278 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 02:00:25.278 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 02:00:25.279 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:00:25.292 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1946'}) 09/01/2025 02:00:27.916 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:27.917 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:27.917 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:27.917 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:27.917 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:27.917 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:27.917 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:27.918 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:27.918 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.918 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:27.918 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.918 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:27.918 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:27.918 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.919 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:31.920 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:31.920 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:31.920 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:31.920 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:31.920 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:31.920 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:31.921 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:31.921 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.921 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:31.921 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.921 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:31.921 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:31.921 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.923 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:35.924 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:35.924 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:35.924 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:35.925 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.449 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:00:39.450 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:00:39.450 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:00:39.450 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1932 09/01/2025 02:00:39.450 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:00:39.450 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 02:00:39.450 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 02:00:39.451 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1932'})(timeout=120) 09/01/2025 02:00:39.452 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 02:00:39.452 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 02:00:39.454 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 02:00:39.454 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:00:39.459 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1932'}) 09/01/2025 02:00:39.925 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:39.926 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:39.926 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:39.926 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:39.926 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:39.926 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:39.927 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:39.927 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:39.927 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.927 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:39.927 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.927 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:39.927 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:39.927 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.927 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:43.928 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:43.928 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:43.928 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:43.929 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.930 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:47.931 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:47.931 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:47.931 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:47.931 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:47.931 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:47.932 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:47.932 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:47.932 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.932 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:47.932 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.932 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:47.932 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:47.932 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.932 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:51.933 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:51.933 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:51.933 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:51.934 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.934 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:55.935 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:55.936 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:56.377 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 02:00:56.377 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:00:56.377 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:00:56.378 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:00:58.151 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2103 09/01/2025 02:00:58.151 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:00:58.151 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:00:58.152 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:00:59.938 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:59.939 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:59.939 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:59.939 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:59.939 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:59.939 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:59.939 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:59.939 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:59.939 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.940 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:59.940 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.940 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.940 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:59.940 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.939 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:03.941 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:03.941 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:03.941 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:03.941 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:03.941 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:03.941 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:03.942 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:03.942 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.942 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:03.942 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.942 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:03.942 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:03.942 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:04.242 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:01:04.242 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:01:04.242 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 02:01:04.242 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:01:04.242 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 02:01:04.242 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 02:01:04.242 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:01:04.242 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:01:07.942 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:07.943 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:07.944 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.945 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:11.946 7f490d10ca40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:11.946 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:11.946 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:11.946 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:11.946 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:11.946 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:11.947 7f490d10ca40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:11.947 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.947 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:11.947 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.947 7f490d10ca40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:11.947 7f490d10ca40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:11.947 7f490d10ca40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.234 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=1983 09/01/2025 02:01:14.235 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:01:14.235 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 02:01:14.235 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:01:14.235 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 02:01:14.235 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 02:01:14.235 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 02:01:14.235 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:01:14.235 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:01:15.025 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:01:15.026 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:01:15.026 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:01:15.026 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:01:15.947 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:15.948 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:15.948 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:15.948 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:15.948 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:15.948 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:15.948 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:15.949 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:15.949 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:15.949 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:15.949 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:15.949 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:15.949 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:15.949 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.950 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:19.951 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:19.951 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:19.951 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:19.951 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:19.951 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:19.952 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:19.952 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:19.952 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.952 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:19.952 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.952 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:19.952 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:19.952 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.953 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:23.954 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:23.954 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:23.954 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:23.955 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:27.955 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:27.956 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:27.956 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:27.956 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:27.956 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:27.956 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:27.956 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:27.956 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:27.956 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:27.957 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:27.957 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:27.957 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:27.957 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:27.957 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.958 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:31.959 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:31.959 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:31.959 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:31.959 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:31.959 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:31.959 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:31.959 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:31.959 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.960 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:31.960 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.960 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:31.960 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:31.960 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:33.492 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 02:01:33.492 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:01:33.493 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:01:33.493 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:01:34.741 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=1872 09/01/2025 02:01:34.742 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:01:34.742 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:01:34.742 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:01:35.715 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration IngressMonitor 09/01/2025 02:01:35.715 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:01:35.715 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1872 09/01/2025 02:01:35.715 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:01:35.715 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 02:01:35.716 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 02:01:35.716 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1872'})(timeout=120) 09/01/2025 02:01:35.718 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 02:01:35.718 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 02:01:35.718 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 02:01:35.718 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:01:35.728 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1872'}) 09/01/2025 02:01:35.959 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:35.960 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:35.961 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.961 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:39.963 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:39.963 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:39.963 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:39.963 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:39.963 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:39.963 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:39.964 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:39.964 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.964 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:39.964 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.964 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:39.964 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:39.964 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.965 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:43.966 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:43.966 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:43.966 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:43.966 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:43.966 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:43.966 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:43.967 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:43.967 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.967 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:43.967 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.967 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:43.967 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:43.967 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.969 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:47.970 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:47.971 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.972 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:47.972 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:47.972 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.972 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:51.973 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.974 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:51.974 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:51.974 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:52.092 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=1983 09/01/2025 02:01:52.093 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:01:52.093 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:01:52.093 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:01:53.809 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration PodMonitor 09/01/2025 02:01:53.810 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:01:53.810 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1983 09/01/2025 02:01:53.810 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:01:53.810 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 02:01:53.810 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 02:01:53.810 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1983'})(timeout=120) 09/01/2025 02:01:53.811 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 02:01:53.812 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 02:01:53.812 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 02:01:53.812 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:01:53.819 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1983'}) 09/01/2025 02:01:55.974 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:55.975 7f490e2bae00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:55.975 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:55.975 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:55.975 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:55.975 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:55.975 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:55.976 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:55.976 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:55.976 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:55.976 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:55.976 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:55.976 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:55.976 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.991 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2103 09/01/2025 02:01:58.992 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:01:58.992 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:01:58.992 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:01:59.976 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:59.977 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:59.977 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:59.977 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:59.978 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:00.117 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2103 09/01/2025 02:02:00.118 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:02:00.118 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:02:00.118 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:02:01.287 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2103 09/01/2025 02:02:01.327 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:02:01.328 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:02:01.328 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:02:02.706 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration NetworkMonitor 09/01/2025 02:02:02.706 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:02:02.706 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2103 09/01/2025 02:02:02.707 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:02:02.707 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 02:02:02.707 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 02:02:02.707 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2103'})(timeout=120) 09/01/2025 02:02:02.708 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 02:02:02.709 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 02:02:02.709 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:02:02.715 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2103'}) 09/01/2025 02:02:02.811 7f490d022b80 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:02:03.978 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:03.979 7f490d022b80 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:03.980 7f490d022b80 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:04.090 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:02:04.090 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:02:04.090 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:02:04.090 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:02:07.982 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:07.983 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:07.983 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:07.983 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:07.983 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:07.983 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:07.983 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:07.983 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:07.984 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:07.984 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:07.984 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:07.984 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:07.984 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:07.984 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:11.985 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:11.987 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:11.987 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:11.987 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:11.987 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:11.987 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:11.987 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:11.987 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:11.987 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:11.988 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:11.988 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:11.988 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:11.988 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:11.988 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.131 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:02:14.131 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:02:14.131 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:02:14.131 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:02:14.387 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:02:14.388 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:02:14.388 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:02:14.388 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:02:15.273 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:02:15.273 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:02:15.273 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:02:15.273 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:02:15.988 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:15.989 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:15.989 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:15.989 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:15.989 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:15.989 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:15.989 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:15.989 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:15.990 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:15.990 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:15.990 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:15.990 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:15.990 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:15.990 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:16.806 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration EndPointMonitor 09/01/2025 02:02:16.807 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:02:16.807 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1948 09/01/2025 02:02:16.807 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:02:16.807 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 02:02:16.808 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 02:02:16.808 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1948'})(timeout=120) 09/01/2025 02:02:16.809 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 02:02:16.809 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 02:02:16.810 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:02:16.821 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1948'}) 09/01/2025 02:02:19.991 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:19.992 7f490d0ae680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:19.992 7f490d0ae680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:19.992 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:19.992 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:19.993 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:23.993 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:23.994 7f490d0ae680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:23.994 7f490d0ae680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:23.994 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:23.994 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:23.994 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:23.994 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:23.994 7f490d0ae680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:23.995 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:23.995 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:23.995 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:23.995 7f490d0ae680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:23.995 7f490d0ae680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:23.995 7f490d0ae680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:25.293 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:02:25.293 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:02:25.293 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:02:25.294 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1946 09/01/2025 02:02:25.294 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:02:25.294 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 02:02:25.294 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 02:02:25.294 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1946'})(timeout=120) 09/01/2025 02:02:25.295 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 02:02:25.296 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 02:02:25.296 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 02:02:25.296 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:02:25.309 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1946'}) 09/01/2025 02:02:27.995 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:27.996 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:27.996 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:27.996 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:27.997 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:31.998 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:31.999 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:31.999 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:31.999 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:32.000 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.001 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:36.002 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:36.002 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:36.003 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:36.024 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=1872 newResVer=2667 09/01/2025 02:02:36.025 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:02:36.025 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:02:36.025 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:02:39.463 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:02:39.463 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:02:39.464 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:02:39.464 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1932 09/01/2025 02:02:39.464 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:02:39.464 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 02:02:39.464 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 02:02:39.464 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1932'})(timeout=120) 09/01/2025 02:02:39.465 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 02:02:39.466 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 02:02:39.468 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 02:02:39.468 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:02:39.472 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1932'}) 09/01/2025 02:02:40.003 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:40.004 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:40.004 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:40.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.005 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:44.007 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:44.007 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:44.007 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:44.007 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:44.007 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:44.007 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:44.007 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:44.008 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.008 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:44.008 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.008 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:44.008 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:44.008 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.009 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:48.011 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:48.011 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:52.010 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:52.011 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:52.012 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:52.013 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:52.013 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:56.013 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:56.014 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:56.014 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:56.014 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:56.014 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:56.014 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:56.014 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:56.015 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:56.015 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:56.015 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:56.015 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:56.015 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:56.015 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:56.015 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:00.016 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:00.017 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:00.017 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:00.017 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:00.017 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:00.017 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:00.017 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:00.018 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:00.018 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:00.018 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:00.018 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:00.018 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:00.018 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:00.018 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:04.018 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:04.019 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:04.019 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:04.019 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:04.020 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.021 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:08.022 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:08.022 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:08.022 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:08.022 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:08.022 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:08.022 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:08.023 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:08.023 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.023 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:08.023 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.023 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:08.023 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:08.023 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:12.025 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:12.026 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:12.026 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:12.026 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:12.027 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.919 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:03:14.919 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:03:14.919 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 02:03:14.919 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:03:14.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 02:03:14.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 02:03:14.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 02:03:14.919 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:03:14.920 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:03:16.027 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:16.028 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:16.028 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:16.028 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:16.028 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:16.028 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:16.028 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:16.029 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:16.029 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:16.029 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:16.029 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:16.029 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:16.029 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:16.029 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:20.030 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:20.031 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:20.032 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:20.032 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:24.032 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:24.033 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:24.033 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:24.033 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:24.033 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:24.033 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:24.033 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:24.034 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:24.034 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:24.034 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:24.034 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:24.034 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:24.034 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:24.034 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.034 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:28.035 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:28.036 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:28.184 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=1948 09/01/2025 02:03:28.185 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:03:28.185 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:03:28.185 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:03:29.947 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration ServiceMonitor 09/01/2025 02:03:29.948 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:03:29.948 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1948 09/01/2025 02:03:29.948 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:03:29.948 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 02:03:29.948 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 02:03:29.948 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1948'})(timeout=120) 09/01/2025 02:03:29.950 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 02:03:29.950 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 02:03:29.950 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 02:03:29.950 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:03:29.952 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1948'}) 09/01/2025 02:03:32.038 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:32.039 7f490d0794a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:32.039 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:32.039 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:32.039 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:32.039 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:32.039 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:32.039 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:32.039 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:32.040 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:32.040 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:32.040 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:32.040 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:32.040 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:35.647 7f490d0794a0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:03:36.039 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:36.080 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=2667 newResVer=2667 09/01/2025 02:03:36.080 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:03:36.080 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:03:36.081 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:03:36.081 7f490d0794a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:36.081 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:36.081 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:36.081 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:36.081 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:36.081 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:36.081 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:36.082 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:36.082 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:36.082 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:36.082 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:36.082 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:36.082 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.072 7f490d0794a0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:03:39.919 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=1932 newResVer=2752 09/01/2025 02:03:39.920 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:03:39.920 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:03:39.920 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:03:40.081 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:40.087 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:40.087 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:40.087 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:40.087 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:40.087 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:40.087 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:40.088 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:40.088 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:40.088 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:40.088 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:40.088 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:40.088 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:40.088 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:41.661 7f490d0792c0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:03:44.088 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:44.089 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:44.089 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:44.090 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.092 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:48.093 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:48.093 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:48.093 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:48.093 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:48.093 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:48.093 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:48.094 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:48.094 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.094 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:48.094 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.094 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:48.094 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:48.094 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:52.093 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:52.094 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:52.094 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:52.094 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:52.095 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:53.822 7f490d022680 [contrail-kube-manager] [ERROR]: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:03:53.822 7f490d022680 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:03:53.822 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:03:53.822 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=1983 09/01/2025 02:03:53.823 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:03:53.823 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 02:03:53.823 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 02:03:53.823 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1983'})(timeout=120) 09/01/2025 02:03:53.824 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 02:03:53.824 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 02:03:53.824 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 02:03:53.824 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:03:53.836 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '1983'}) 09/01/2025 02:03:56.096 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:56.097 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:56.097 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:56.097 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:56.097 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:56.097 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:56.097 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:56.098 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:56.098 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.098 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:56.098 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.098 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:56.098 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:56.098 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:00.099 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:00.100 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:00.100 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:00.100 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:00.101 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.717 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:04:02.717 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:04:02.717 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:04:02.717 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2103 09/01/2025 02:04:02.717 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:04:02.718 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 02:04:02.718 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 02:04:02.718 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2103'})(timeout=120) 09/01/2025 02:04:02.719 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 02:04:02.719 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 02:04:02.720 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:04:02.733 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2103'}) 09/01/2025 02:04:04.101 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:04.102 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:04.102 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:04.102 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:04.103 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:04.104 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.104 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:08.105 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.106 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:08.107 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:08.107 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:12.109 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:12.110 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:12.110 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:12.110 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:12.110 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:12.110 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:12.110 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:12.111 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:12.111 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:12.111 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:12.111 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:12.111 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:12.111 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:12.111 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.111 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:16.112 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:16.112 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:16.112 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:16.112 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:16.112 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:16.112 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:16.113 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:16.113 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.113 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:16.113 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.113 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.113 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:16.113 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:16.168 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=1948 newResVer=2776 09/01/2025 02:04:16.168 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:04:16.168 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:04:16.168 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:04:20.113 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:20.114 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:20.114 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:20.115 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:24.117 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:24.118 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:24.118 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:24.118 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:24.118 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:24.118 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:24.118 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:24.119 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:24.119 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:24.119 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:24.119 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:24.119 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:24.119 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:24.119 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:25.068 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=1946 newResVer=2774 09/01/2025 02:04:25.069 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:04:25.069 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:04:25.069 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:04:28.119 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:28.120 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:28.120 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:28.120 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:28.120 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:28.120 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:28.120 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:28.121 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:28.121 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:28.121 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:28.121 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:28.121 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:28.121 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:28.121 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:29.642 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=1948 newResVer=2778 09/01/2025 02:04:29.642 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:04:29.642 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:04:29.642 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:04:32.122 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:32.123 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:32.123 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:32.123 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:32.123 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:32.123 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:32.124 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:32.124 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:32.124 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:32.124 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:32.124 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:32.124 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:32.124 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:32.124 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:35.275 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=2667 newResVer=2667 09/01/2025 02:04:35.275 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:04:35.275 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:04:35.275 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:04:36.124 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:36.125 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:36.125 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:36.125 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:36.126 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:39.650 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=2752 09/01/2025 02:04:39.650 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:04:39.650 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:04:39.650 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:04:40.126 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:40.127 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:40.128 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:40.128 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:40.128 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:40.128 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:40.128 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:40.128 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:40.128 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:40.128 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:40.129 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:40.129 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:40.129 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:40.129 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:44.129 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:44.130 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:44.130 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:44.130 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:44.130 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:44.130 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:44.130 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:44.130 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:44.131 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:44.131 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:44.131 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:44.131 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:44.131 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:44.131 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.132 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:48.133 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:48.134 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:48.135 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:52.134 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:52.136 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:52.137 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:52.137 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:52.137 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:52.137 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:53.492 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=1983 newResVer=2826 09/01/2025 02:04:53.493 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:04:53.493 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:04:53.493 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:04:56.137 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:56.138 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:56.138 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:56.138 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:56.139 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:00.141 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:00.142 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:00.142 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:00.142 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:00.143 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:04.143 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:04.144 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:04.144 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:04.144 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:04.144 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:04.144 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:04.145 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:04.145 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:04.145 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:04.145 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:04.145 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:04.145 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:04.145 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:04.145 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.145 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:08.146 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:08.146 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:08.146 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:08.146 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:08.146 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:08.147 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:08.147 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:08.147 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.147 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:08.147 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.147 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:08.147 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:08.147 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:12.149 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:12.150 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:12.151 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:12.151 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:12.151 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:12.151 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:12.151 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:12.151 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:12.151 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:12.152 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:12.152 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:12.152 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:12.152 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:12.152 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:16.151 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:16.152 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:16.152 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:16.152 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:16.152 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:16.152 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:16.153 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:16.153 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:16.153 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:16.153 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:16.153 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:16.153 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:16.153 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:16.153 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:16.280 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=2776 newResVer=2776 09/01/2025 02:05:16.280 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:05:16.280 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:05:16.280 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:05:20.154 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:20.155 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:20.156 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:24.158 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:24.159 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:24.160 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:24.161 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:24.161 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:25.235 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=2774 newResVer=2774 09/01/2025 02:05:25.235 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:05:25.236 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:05:25.236 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:05:28.160 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:28.161 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:28.161 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:28.161 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:28.161 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:28.162 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:29.522 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=2778 newResVer=2778 09/01/2025 02:05:29.522 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:05:29.522 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:05:29.522 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:05:32.163 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:32.164 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:32.164 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:32.164 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:32.164 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:32.164 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:32.164 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:32.165 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:32.165 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:32.165 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:32.165 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:32.165 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:32.165 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:32.165 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:36.074 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=2667 newResVer=2667 09/01/2025 02:05:36.074 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:05:36.075 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 02:05:36.075 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:05:36.075 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 02:05:36.075 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 02:05:36.075 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 02:05:36.075 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:05:36.076 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:05:36.165 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:36.166 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:36.166 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:36.167 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.435 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=2752 09/01/2025 02:05:39.435 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:05:39.435 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:05:39.435 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:05:40.167 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:40.168 7f490d155180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:40.168 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:40.168 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:40.169 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:44.169 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:44.170 7f490d155180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:44.170 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:44.170 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:44.170 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:44.170 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:44.170 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:44.171 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:44.171 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:44.171 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:44.171 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:44.171 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:44.171 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:44.171 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:48.172 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:48.173 7f490d155180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:48.173 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:48.173 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:48.173 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:48.173 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:48.173 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:48.174 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:48.174 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:48.174 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:48.174 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:48.174 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:48.174 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:48.174 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:52.174 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:52.175 7f490d155180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:52.175 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:52.175 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:52.175 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:52.175 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:52.176 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:52.176 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:52.176 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:52.176 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:52.176 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:52.176 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:52.176 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:52.176 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:53.631 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=2826 09/01/2025 02:05:53.632 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:05:53.632 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:05:53.632 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:05:56.177 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:56.178 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:56.179 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:56.179 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:56.179 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:56.179 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:00.178 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:00.179 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:00.180 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.392 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2103 newResVer=2897 09/01/2025 02:06:02.392 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:06:02.392 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:06:02.393 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:06:04.182 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:04.183 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:04.183 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:04.183 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:04.183 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:04.183 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:04.183 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:04.183 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:04.183 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:04.184 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:04.184 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:04.184 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:04.184 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:04.184 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.184 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:08.185 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:08.185 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:08.185 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:08.185 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:08.186 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:12.188 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:12.189 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:12.189 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:12.189 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:12.189 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:12.189 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:12.189 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:12.190 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:12.190 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:12.190 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:12.190 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:12.190 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:12.190 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:12.190 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:16.190 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:16.191 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:16.192 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:16.192 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:16.192 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:16.192 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:17.011 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=2776 newResVer=2776 09/01/2025 02:06:17.011 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:06:17.011 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 02:06:17.011 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:06:17.011 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 02:06:17.011 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 02:06:17.011 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:06:17.011 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:06:20.192 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:20.194 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:20.195 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:20.195 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:20.195 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:24.195 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:24.196 7f490d0794a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:24.196 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:24.196 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:24.196 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:24.196 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:24.197 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:24.197 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:24.197 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:24.197 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:24.197 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:24.197 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:24.197 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:24.197 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:25.965 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=2774 newResVer=2774 09/01/2025 02:06:25.965 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:06:25.965 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 02:06:25.965 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:06:25.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 02:06:25.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 02:06:25.965 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 02:06:25.966 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:06:25.966 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:06:28.198 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:28.199 7f490d10c5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:28.200 7f490d10c5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:28.200 7f490d10c5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.070 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=2778 newResVer=2778 09/01/2025 02:06:30.070 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:06:30.071 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:06:30.071 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:06:32.200 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:32.202 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:32.202 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:32.202 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:32.202 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:32.202 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:32.202 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:32.202 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:32.203 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:32.203 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:32.203 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:32.203 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:32.203 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:32.203 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:36.046 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=2667 newResVer=2667 09/01/2025 02:06:36.047 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:06:36.047 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:06:36.047 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:06:36.202 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:36.203 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:36.204 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:36.205 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:36.205 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:37.851 7f490d10c220 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:06:40.117 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=2752 09/01/2025 02:06:40.117 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:06:40.117 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 02:06:40.117 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:06:40.117 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 02:06:40.118 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 02:06:40.118 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:06:40.120 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 02:06:40.120 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:06:40.204 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:40.205 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:40.205 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:40.205 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:40.205 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:40.205 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:40.206 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:40.206 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:40.206 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:40.206 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:40.206 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:40.206 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:40.206 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:40.206 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:44.207 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:44.208 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:44.208 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:44.208 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:44.208 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:44.208 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:44.208 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:44.208 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:44.209 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:44.209 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:44.209 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:44.209 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:44.209 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:44.209 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.210 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:48.211 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:48.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:52.212 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:52.214 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:52.214 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:52.214 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:52.214 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:52.214 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:52.214 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:52.214 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:52.215 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:52.215 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:52.215 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:52.215 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:52.215 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:52.215 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.052 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=2826 09/01/2025 02:06:54.052 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:06:54.052 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:06:54.052 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:06:56.214 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:56.215 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:56.215 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:56.215 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:56.215 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:56.215 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:56.216 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:56.216 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:56.216 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:56.216 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:56.216 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:56.216 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:56.216 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:56.216 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:00.218 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:00.219 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:00.219 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:00.219 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:00.219 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:00.219 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:00.219 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:00.220 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:00.220 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:00.220 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:00.220 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:00.220 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:00.220 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:00.220 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:01.942 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2897 newResVer=2897 09/01/2025 02:07:01.942 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:07:01.942 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:07:01.942 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:07:04.220 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:04.227 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:04.228 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.229 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:08.230 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:08.231 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.232 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:08.232 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:08.232 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:12.232 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:12.233 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:12.234 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:12.234 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:12.234 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:12.234 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:12.234 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:12.234 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:12.234 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:12.234 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:12.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:12.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:12.235 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:12.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.234 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:16.235 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:16.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:16.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:16.235 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:16.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:16.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:16.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:16.235 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.236 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:16.236 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.236 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.236 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:16.236 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:16.979 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=2776 newResVer=2776 09/01/2025 02:07:16.980 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:07:16.980 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:07:16.980 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:07:20.237 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:20.238 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:20.238 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:20.238 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:20.238 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:20.238 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:20.238 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:20.238 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:20.239 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:20.239 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:20.239 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:20.239 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:20.239 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:20.239 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:24.241 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:24.242 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:24.242 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:24.242 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:24.242 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:24.242 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:24.242 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:24.243 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:24.243 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:24.243 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:24.243 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:24.243 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:24.243 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:24.243 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:25.116 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=2774 newResVer=2774 09/01/2025 02:07:25.117 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:07:25.117 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:07:25.117 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:07:28.243 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:28.244 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:28.244 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:28.244 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:28.244 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:28.244 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:28.244 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:28.244 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:28.245 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:28.245 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:28.245 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:28.245 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:28.245 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:28.245 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:29.552 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=2778 newResVer=2778 09/01/2025 02:07:29.552 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:07:29.552 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:07:29.552 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:07:32.245 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:32.246 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:32.246 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:32.246 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:32.247 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.604 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=2667 newResVer=2667 09/01/2025 02:07:35.604 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:07:35.605 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:07:35.605 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:07:36.247 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:36.248 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:36.248 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:36.248 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:36.248 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:36.248 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:36.249 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:36.249 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:36.249 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:36.249 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:36.249 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:36.249 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:36.249 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:36.249 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.290 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=2752 09/01/2025 02:07:39.290 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:07:39.290 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:07:39.290 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:07:40.249 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:40.250 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:40.250 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:40.250 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:40.250 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:40.250 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:40.250 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:40.251 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:40.251 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:40.251 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:40.251 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:40.251 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:40.251 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:40.251 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:44.251 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:44.252 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:44.252 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:44.252 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:44.252 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:44.252 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:44.252 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:44.252 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:44.252 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:44.253 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:44.253 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:44.253 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:44.253 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:44.253 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:48.254 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:48.255 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:48.255 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:48.255 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:48.255 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:48.255 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:48.255 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:48.256 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:48.256 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:48.256 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:48.256 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:48.256 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:48.256 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:48.256 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:52.256 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:52.258 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:52.259 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:52.259 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:52.259 7f490d0ae5e0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:52.259 7f490d0ae5e0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:53.167 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=2826 09/01/2025 02:07:53.167 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:07:53.168 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:07:53.168 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:07:56.259 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:56.260 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:56.260 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:56.260 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:56.261 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:00.262 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:00.263 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:00.263 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:00.264 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:00.264 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:00.264 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:00.264 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:00.264 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:00.264 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:00.264 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:00.265 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:00.265 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:00.265 7f490d0ae9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:00.265 7f490d0ae9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.933 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2897 newResVer=2897 09/01/2025 02:08:02.933 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:08:02.933 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 02:08:02.933 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:08:02.933 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 02:08:02.933 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 02:08:02.933 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:08:02.933 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:08:04.264 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:04.265 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:04.265 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:04.265 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:04.265 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:04.266 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.267 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:08.268 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:08.268 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:08.268 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:08.269 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:12.269 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:12.270 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:12.271 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.272 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:16.273 7f490d0aea40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:16.273 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:16.273 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:16.273 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:16.273 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:16.273 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:16.273 7f490d0aea40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:16.274 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.274 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:16.274 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.274 7f490d0aea40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.274 7f490d0aea40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:16.274 7f490d0aea40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:16.282 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=2776 newResVer=2776 09/01/2025 02:08:16.282 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:08:16.282 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:08:16.282 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:08:20.274 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:20.276 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:20.277 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:20.277 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:20.277 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:20.277 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:24.276 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:24.277 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:24.277 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:24.277 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:24.277 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:24.278 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:25.711 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=2774 newResVer=2774 09/01/2025 02:08:25.711 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:08:25.712 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:08:25.712 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:08:28.278 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:28.279 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:28.279 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:28.279 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:28.280 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:28.280 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:28.280 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:28.280 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:28.280 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:28.280 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:28.280 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:28.280 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:28.281 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:28.281 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.132 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=2778 newResVer=2778 09/01/2025 02:08:30.132 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:08:30.132 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 02:08:30.132 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:08:30.132 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 02:08:30.133 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 02:08:30.133 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 02:08:30.133 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:08:30.133 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:08:32.281 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:32.282 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:32.282 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:32.282 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:32.282 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:32.282 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:32.283 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:32.283 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:32.283 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:32.283 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:32.283 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:32.283 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:32.283 7f490d0ed9a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:32.283 7f490d0ed9a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:36.180 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=2667 newResVer=2667 09/01/2025 02:08:36.180 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:08:36.180 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:08:36.181 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:08:36.283 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:36.284 7f490d155180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:36.285 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:36.286 7f490d155180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:36.286 7f490d155180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:36.286 7f490d155180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.512 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=2752 09/01/2025 02:08:39.512 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:08:39.512 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:08:39.512 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:08:40.285 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:40.286 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:40.287 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:40.288 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:40.288 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:44.287 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:44.288 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:44.288 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:44.288 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:44.289 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.437 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=2667 newResVer=2667 09/01/2025 02:08:46.437 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:08:46.437 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:08:46.437 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:08:47.589 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration IngressMonitor 09/01/2025 02:08:47.589 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:08:47.589 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2667 09/01/2025 02:08:47.589 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:08:47.589 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 02:08:47.590 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 02:08:47.590 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2667'})(timeout=120) 09/01/2025 02:08:47.591 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 02:08:47.591 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 02:08:47.591 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 02:08:47.591 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:08:47.602 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2667'}) 09/01/2025 02:08:48.289 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:48.290 7f490d13fcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:48.290 7f490d13fcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:48.290 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:48.291 7f490d13fcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:52.291 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:52.292 7f490d13fcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:52.292 7f490d13fcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:52.292 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:52.293 7f490d13fcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:52.293 7f490d13fcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:52.293 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:52.293 7f490d13fcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:52.293 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:52.293 7f490d13fcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:52.293 7f490d13fcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:52.293 7f490d13fcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:52.294 7f490d13fcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:52.294 7f490d13fcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:53.345 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=2826 09/01/2025 02:08:53.346 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:08:53.346 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 02:08:53.346 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:08:53.346 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 02:08:53.346 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 02:08:53.346 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 02:08:53.346 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:08:53.346 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:08:56.294 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:56.295 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:56.296 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:56.296 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:56.296 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:56.296 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:00.298 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:00.299 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:00.299 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:00.299 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:00.299 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:00.299 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:00.299 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:00.300 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:00.300 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:00.300 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:00.300 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:00.300 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:00.300 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:00.300 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.907 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2897 newResVer=2897 09/01/2025 02:09:02.907 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:09:02.907 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:09:02.907 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:09:04.300 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:04.301 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:04.301 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:04.301 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:04.301 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:04.301 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:04.301 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:04.302 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:04.302 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:04.302 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:04.302 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:04.302 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:04.302 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:04.302 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.303 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:08.304 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:08.304 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:08.304 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:08.304 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:08.304 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:08.304 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:08.304 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:08.305 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.305 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:08.305 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.305 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:08.305 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:08.305 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:12.305 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:12.306 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:12.306 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:12.306 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:12.306 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:12.306 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:12.306 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:12.307 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:12.307 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:12.307 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:12.307 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:12.307 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:12.307 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:12.307 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.307 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:16.308 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:16.308 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:16.308 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:16.308 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:16.309 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:16.846 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=2776 newResVer=2776 09/01/2025 02:09:16.846 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:09:16.846 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:09:16.846 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:09:20.308 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:20.309 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:20.310 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:20.311 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:20.311 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:20.922 7f490d10c180 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:09:23.351 7f490d10c180 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:09:24.310 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:24.311 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:24.311 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:24.312 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.072 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=2774 newResVer=2774 09/01/2025 02:09:26.072 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:09:26.072 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:09:26.072 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:09:26.884 7f490d10c220 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:09:28.312 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:28.313 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:28.313 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:28.313 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:28.313 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:28.313 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:28.313 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:28.314 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:28.314 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:28.314 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:28.314 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:28.314 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:28.314 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:28.314 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.138 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=2778 newResVer=2778 09/01/2025 02:09:30.139 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:09:30.139 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:09:30.139 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:09:32.314 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:32.315 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:32.316 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:36.315 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:36.316 7f490e2bae00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:36.316 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:36.316 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:36.317 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.589 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=2752 09/01/2025 02:09:39.589 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:09:39.589 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:09:39.589 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:09:40.318 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:40.319 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:40.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:40.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:40.320 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:40.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:44.320 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:44.321 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:44.321 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:44.321 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:44.321 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:44.321 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:44.321 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:44.321 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:44.322 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:44.322 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:44.322 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:44.322 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:44.322 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:44.322 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:48.323 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:48.324 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:48.324 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:48.325 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:48.326 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:52.325 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:52.326 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:52.327 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.264 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=2826 09/01/2025 02:09:54.264 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:09:54.264 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:09:54.264 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:09:56.327 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:56.328 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:56.328 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:56.328 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:56.329 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:57.028 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2897 newResVer=2897 09/01/2025 02:09:57.028 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:09:57.028 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:09:57.028 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:09:58.040 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2897 newResVer=2897 09/01/2025 02:09:58.040 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:09:58.040 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:09:58.040 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:09:59.922 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration NetworkMonitor 09/01/2025 02:09:59.923 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:09:59.923 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2897 09/01/2025 02:09:59.923 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:09:59.923 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 02:09:59.923 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 02:09:59.923 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2897'})(timeout=120) 09/01/2025 02:09:59.924 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 02:09:59.925 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 02:09:59.925 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:09:59.942 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2897'}) 09/01/2025 02:10:00.329 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:00.331 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:00.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:00.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:00.331 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:00.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:00.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:00.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:00.331 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:00.332 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:00.332 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:00.332 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:00.332 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:00.332 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:04.332 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:04.333 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:04.333 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:04.333 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:04.333 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:04.333 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:04.334 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:04.334 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:04.334 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:04.334 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:04.334 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:04.334 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:04.334 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:04.334 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:05.053 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=2774 newResVer=2774 09/01/2025 02:10:05.054 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:10:05.054 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:10:05.054 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:10:06.232 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration NamespaceMonitor 09/01/2025 02:10:06.232 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:10:06.232 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2774 09/01/2025 02:10:06.232 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:10:06.233 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 02:10:06.233 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 02:10:06.233 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2774'})(timeout=120) 09/01/2025 02:10:06.234 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 02:10:06.234 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 02:10:06.234 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 02:10:06.234 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:10:06.245 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2774'}) 09/01/2025 02:10:08.335 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:08.336 7f490d0edcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:08.336 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:08.336 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:08.336 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:08.336 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:08.336 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:08.337 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:08.337 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:08.337 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:08.337 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:08.337 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:08.337 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:08.337 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:12.339 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:12.343 7f490d0edcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:12.343 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:12.343 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:12.343 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:12.343 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:12.343 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:12.344 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:12.344 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:12.344 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:12.344 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:12.344 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:12.344 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:12.344 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.993 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=2776 newResVer=2776 09/01/2025 02:10:15.994 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:10:15.994 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:10:15.994 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:10:16.344 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:16.345 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:16.345 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:16.345 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:16.345 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:16.345 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:16.345 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:16.346 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:16.346 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:16.346 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:16.346 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:16.346 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:16.346 7f490d0ed7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:16.346 7f490d0ed7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:17.467 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration EndPointMonitor 09/01/2025 02:10:17.467 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:10:17.467 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2776 09/01/2025 02:10:17.468 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:10:17.468 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 02:10:17.468 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 02:10:17.468 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2776'})(timeout=120) 09/01/2025 02:10:17.469 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 02:10:17.469 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 02:10:17.469 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:10:17.478 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2776'}) 09/01/2025 02:10:19.378 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=2752 09/01/2025 02:10:19.378 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:10:19.379 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:10:19.379 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:10:20.347 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:20.348 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:20.348 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:20.348 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:20.348 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:20.348 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:20.348 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:20.349 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:20.349 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:20.349 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:20.349 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:20.349 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:20.349 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:20.349 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:20.616 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=2752 09/01/2025 02:10:20.616 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:10:20.616 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:10:20.616 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:10:21.534 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration NetworkPolicyMonitor 09/01/2025 02:10:21.535 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:10:21.535 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2752 09/01/2025 02:10:21.535 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:10:21.535 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 02:10:21.535 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 02:10:21.535 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2752'})(timeout=120) 09/01/2025 02:10:21.536 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 02:10:21.536 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 02:10:21.538 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 02:10:21.539 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:10:21.539 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2752'}) 09/01/2025 02:10:22.168 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=2778 newResVer=2778 09/01/2025 02:10:22.169 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:10:22.169 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:10:22.169 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:10:23.747 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration ServiceMonitor 09/01/2025 02:10:23.747 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:10:23.747 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2778 09/01/2025 02:10:23.748 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:10:23.748 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 02:10:23.748 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 02:10:23.748 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2778'})(timeout=120) 09/01/2025 02:10:23.749 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 02:10:23.749 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 02:10:23.749 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 02:10:23.749 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:10:23.755 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2778'}) 09/01/2025 02:10:24.349 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:24.350 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:24.350 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:24.350 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:24.350 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:24.350 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:24.350 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:24.351 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:24.351 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:24.351 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:24.351 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:24.351 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:24.351 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:24.351 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.351 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:28.352 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:28.352 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:28.352 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:28.353 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:32.353 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:32.354 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:32.354 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:32.354 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:32.354 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:32.354 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:32.354 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:32.355 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:32.355 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:32.355 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:32.355 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:32.355 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:32.355 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:32.355 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:36.354 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:36.356 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:36.357 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:36.357 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:36.357 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:36.357 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:40.357 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:40.358 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:40.359 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:40.359 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:44.359 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:44.361 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:44.362 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:44.362 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.605 7f490d079ae0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:10:47.605 7f490d079ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:10:47.605 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:10:47.605 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2667 09/01/2025 02:10:47.606 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:10:47.606 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 02:10:47.606 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 02:10:47.606 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2667'})(timeout=120) 09/01/2025 02:10:47.607 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 02:10:47.607 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 02:10:47.608 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 02:10:47.608 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:10:47.620 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2667'}) 09/01/2025 02:10:48.362 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:48.363 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:48.363 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:48.363 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:48.363 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:48.363 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:48.363 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:48.364 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:48.364 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:48.364 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:48.364 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:48.364 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:48.364 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:48.364 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:52.364 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:52.365 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:52.365 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:52.365 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:52.366 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.987 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=2826 09/01/2025 02:10:54.987 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:10:54.987 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:10:54.987 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:10:56.367 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:56.368 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:56.369 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:56.369 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:56.369 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:56.369 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:56.369 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:56.369 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:56.369 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:56.369 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:56.370 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:56.370 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:56.370 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:56.370 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:00.370 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:00.371 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:00.371 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:00.371 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:00.371 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:00.371 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:00.371 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:00.372 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:00.372 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:00.372 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:00.372 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:00.372 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:00.372 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:00.372 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:04.372 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:04.373 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:04.373 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:04.373 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:04.373 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:04.373 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:04.373 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:04.374 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:04.374 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:04.374 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:04.374 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:04.374 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:04.374 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:04.374 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.374 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:08.375 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.376 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:08.377 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:08.377 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:12.377 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:12.378 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:12.378 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:12.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:16.379 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:16.380 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:16.381 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:16.382 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:20.383 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:20.384 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:20.384 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:20.384 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:20.384 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:20.384 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:20.384 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:20.384 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:20.385 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:20.385 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:20.385 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:20.385 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:20.385 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:20.385 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:24.386 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:24.387 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:24.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:24.389 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:24.389 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.158 7f490d0cc4a0 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:11:28.388 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:28.389 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:28.389 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:28.389 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:28.389 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:28.389 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:28.389 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:28.390 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:28.390 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:28.390 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:28.390 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:28.390 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:28.390 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:28.390 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:32.390 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:32.392 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:32.393 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:32.393 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:32.393 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:32.393 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:36.394 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:36.395 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:36.395 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:36.396 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:36.396 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:36.396 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:36.396 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:36.396 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:36.396 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:36.396 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:36.396 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:36.397 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:36.397 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:36.397 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:40.396 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:40.397 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:40.397 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:40.398 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:40.399 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:44.399 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:44.400 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:44.400 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:44.400 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:44.400 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:44.400 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:44.400 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:44.400 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:44.400 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:44.401 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:44.401 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:44.401 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:44.401 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:44.401 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:48.401 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:48.402 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:48.402 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:48.402 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:48.402 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:48.402 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:48.402 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:48.403 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:48.403 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:48.403 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:48.403 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:48.403 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:48.403 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:48.403 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:52.403 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:52.404 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:52.405 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:52.405 7f490d0cc4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:52.405 7f490d0cc4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.200 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=2826 09/01/2025 02:11:54.200 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:11:54.201 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:11:54.201 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:11:56.405 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:56.407 7f490d0cce00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:56.407 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:56.407 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:56.407 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:56.407 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:56.407 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:56.408 7f490d0cce00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:56.408 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:56.408 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:56.408 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:56.408 7f490d0cce00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:56.408 7f490d0cce00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:56.408 7f490d0cce00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.944 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:11:59.944 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:11:59.944 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:11:59.944 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2897 09/01/2025 02:11:59.944 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:11:59.945 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 02:11:59.945 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 02:11:59.945 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2897'})(timeout=120) 09/01/2025 02:11:59.946 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 02:11:59.947 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 02:11:59.947 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:11:59.961 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2897'}) 09/01/2025 02:12:00.408 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:00.409 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:00.409 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:00.410 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:00.410 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:00.410 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:00.410 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:00.410 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:00.410 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:00.410 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:00.411 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:00.411 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:00.411 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:00.411 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.410 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:04.411 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:04.412 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.413 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:04.413 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:04.413 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:06.248 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:06.248 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:06.248 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:12:06.248 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2774 09/01/2025 02:12:06.248 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:12:06.248 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 02:12:06.249 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 02:12:06.249 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2774'})(timeout=120) 09/01/2025 02:12:06.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 02:12:06.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 02:12:06.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 02:12:06.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:12:06.260 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2774'}) 09/01/2025 02:12:08.412 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:08.414 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:08.414 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:08.414 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:08.414 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:08.414 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:08.414 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:08.414 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:08.414 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.415 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:08.415 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.415 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:08.415 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:08.415 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.415 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:12.416 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:12.416 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:12.416 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:12.416 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:12.416 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:12.416 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:12.417 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:12.417 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.417 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:12.417 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.417 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:12.417 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:12.417 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.417 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:16.418 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:16.419 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:16.419 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:16.419 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:16.419 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:16.419 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:16.419 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:16.420 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.420 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:16.420 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.420 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:16.420 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:16.420 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:17.480 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:17.480 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:17.480 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:12:17.480 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2776 09/01/2025 02:12:17.481 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:12:17.481 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 02:12:17.481 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 02:12:17.481 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2776'})(timeout=120) 09/01/2025 02:12:17.482 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 02:12:17.483 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 02:12:17.483 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:12:17.494 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2776'}) 09/01/2025 02:12:20.419 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:20.420 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:20.420 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:20.420 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:20.420 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:20.420 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:20.420 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:20.421 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:20.421 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:20.421 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:20.421 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:20.421 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:20.421 7f490d0ae4a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:20.421 7f490d0ae4a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:21.542 7f490d0222c0 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:21.542 7f490d0222c0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:21.542 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:12:21.542 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2752 09/01/2025 02:12:21.542 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:12:21.543 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Connect Kube API result 0 09/01/2025 02:12:21.543 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 02:12:21.543 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2752'})(timeout=120) 09/01/2025 02:12:21.544 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 02:12:21.544 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 02:12:21.546 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 02:12:21.546 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:12:21.554 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/networkpolicies ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2752'}) 09/01/2025 02:12:23.756 7f490d0224a0 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:23.757 7f490d0224a0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:23.757 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:12:23.757 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2778 09/01/2025 02:12:23.757 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:12:23.758 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 02:12:23.758 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 02:12:23.758 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2778'})(timeout=120) 09/01/2025 02:12:23.759 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 02:12:23.760 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 02:12:23.760 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 02:12:23.760 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:12:23.768 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2778'}) 09/01/2025 02:12:24.421 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:24.422 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:24.422 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:24.423 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:24.424 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:24.424 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.423 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:28.424 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:28.424 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:28.424 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:28.425 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:28.425 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:28.425 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:28.425 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:28.425 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.425 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:28.425 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.425 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:28.426 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:28.426 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:29.767 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=2826 09/01/2025 02:12:29.767 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:12:29.767 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:12:29.768 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:12:31.488 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: stop iteration PodMonitor 09/01/2025 02:12:31.488 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:12:31.488 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2826 09/01/2025 02:12:31.488 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:12:31.488 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 02:12:31.488 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 02:12:31.489 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2826'})(timeout=120) 09/01/2025 02:12:31.490 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 02:12:31.490 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 02:12:31.490 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 02:12:31.490 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:12:31.501 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2826'}) 09/01/2025 02:12:32.426 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:32.427 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:32.427 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:32.427 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:32.427 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:32.427 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:32.427 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:32.428 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:32.428 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:32.428 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:32.428 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:32.428 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:32.428 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:32.428 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.428 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:36.429 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:36.429 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:36.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.430 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:40.431 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:40.431 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:40.431 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:40.431 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:40.431 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:40.431 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:40.432 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:40.432 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.432 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:40.432 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.432 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:40.432 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:40.432 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.433 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:44.434 7f490d0cccc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:44.434 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:44.434 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:44.434 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:44.434 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:44.434 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:44.435 7f490d0cccc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:44.435 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.435 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:44.435 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.435 7f490d0cccc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.435 7f490d0cccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:44.435 7f490d0cccc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.622 7f490d079ae0 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:47.622 7f490d079ae0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:12:47.622 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:12:47.622 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2667 09/01/2025 02:12:47.623 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:12:47.623 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Connect Kube API result 0 09/01/2025 02:12:47.623 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 02:12:47.623 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Start Watching request https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2667'})(timeout=120) 09/01/2025 02:12:47.624 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 02:12:47.624 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 02:12:47.625 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 02:12:47.625 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:12:47.637 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Watches https://10.0.0.50:6443/apis/networking.k8s.io/v1/ingresses ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2667'}) 09/01/2025 02:12:48.435 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:48.436 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:48.437 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.438 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.438 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:48.438 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.438 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:52.439 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:52.439 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:52.439 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:52.440 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:52.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:52.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:52.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:52.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.440 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:52.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.440 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.441 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:52.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.441 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:56.442 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:56.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:56.442 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:56.442 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:56.443 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.444 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:00.445 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:00.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:00.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:00.445 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:00.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:00.445 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:00.446 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:00.446 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.446 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:00.446 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.446 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.446 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:00.446 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.446 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:04.447 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:04.447 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:04.447 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:04.447 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:04.448 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.449 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:08.450 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:08.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:08.450 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:08.450 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:08.451 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.452 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:12.453 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:12.453 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:12.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.454 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:16.455 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:16.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:16.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:16.455 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:16.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:16.455 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:16.456 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:16.456 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.456 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:16.456 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.456 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.456 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:16.456 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.457 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:20.459 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:20.459 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:20.459 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:20.459 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:20.459 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:20.459 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:20.459 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:20.459 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.460 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:20.460 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.460 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.460 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:20.460 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.460 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:24.461 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.462 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.463 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:24.463 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.463 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:28.464 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:28.464 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:28.464 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:28.465 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.466 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:32.467 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:32.467 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:32.467 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:32.467 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:32.467 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:32.467 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:32.467 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:32.468 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.468 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:32.468 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.468 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.468 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:32.468 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.469 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:36.471 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:36.471 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:36.471 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:36.471 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:36.471 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:36.471 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:36.472 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:36.472 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.472 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:36.472 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.472 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.472 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:36.472 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.472 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:40.473 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:40.473 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:40.473 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:40.473 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:40.473 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:40.473 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:40.474 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:40.474 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.474 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:40.474 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.474 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.474 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:40.474 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.474 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:44.475 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.476 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.476 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:44.476 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.205 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=2667 newResVer=3453 09/01/2025 02:13:48.205 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:13:48.205 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:13:48.205 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:13:48.475 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:48.476 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:48.476 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:48.476 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:48.476 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:48.476 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:48.477 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:48.477 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:48.477 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.477 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:48.477 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.477 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.477 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:48.477 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.477 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.478 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:52.479 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.479 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.479 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:52.479 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.480 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:56.481 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:56.481 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:56.481 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:56.481 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:56.481 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:56.481 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:56.482 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:56.482 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.482 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:56.482 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.482 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:56.482 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:56.482 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.963 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:13:59.964 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:13:59.964 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:13:59.964 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2897 09/01/2025 02:13:59.964 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:13:59.964 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 02:13:59.964 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 02:13:59.964 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2897'})(timeout=120) 09/01/2025 02:13:59.966 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 02:13:59.966 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 02:13:59.966 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:13:59.979 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2897'}) 09/01/2025 02:14:00.482 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:00.483 7f490d0edcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:00.483 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:00.483 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.484 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:00.485 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.484 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:04.485 7f490d0edcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:04.485 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:04.485 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:04.485 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:04.485 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:04.485 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:04.485 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:04.486 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.486 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:04.486 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.486 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.486 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:04.486 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:06.263 7f490d022a40 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:14:06.263 7f490d022a40 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:14:06.263 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:14:06.263 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2774 09/01/2025 02:14:06.263 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:14:06.264 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Connect Kube API result 0 09/01/2025 02:14:06.264 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 02:14:06.264 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2774'})(timeout=120) 09/01/2025 02:14:06.265 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 02:14:06.265 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 02:14:06.266 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 02:14:06.266 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:14:06.278 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Watches https://10.0.0.50:6443/api/v1/namespaces ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2774'}) 09/01/2025 02:14:08.486 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:08.487 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.488 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:08.489 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.490 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:12.491 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:12.491 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:12.491 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:12.491 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:12.492 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.493 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:16.494 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:16.494 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:16.494 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:16.494 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:16.494 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:16.494 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:16.494 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:16.494 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.495 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:16.495 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.495 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.495 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:16.495 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:17.496 7f490d079ea0 [contrail-kube-manager] [ERROR]: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:14:17.496 7f490d079ea0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError EndPointMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:14:17.496 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:14:17.496 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2776 09/01/2025 02:14:17.496 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:14:17.497 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Connect Kube API result 0 09/01/2025 02:14:17.497 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 02:14:17.497 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Start Watching request https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2776'})(timeout=120) 09/01/2025 02:14:17.499 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 02:14:17.499 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 02:14:17.499 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:14:17.512 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Watches https://10.0.0.50:6443/api/v1/endpoints ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2776'}) 09/01/2025 02:14:20.495 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:20.531 7f490d0ae540 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:20.531 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:20.531 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:20.531 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:20.531 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:20.531 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:20.532 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:20.532 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.532 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:20.532 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.532 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.532 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:20.532 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:21.532 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=2752 newResVer=3568 09/01/2025 02:14:21.532 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:14:21.532 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:14:21.532 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:14:23.770 7f490d0224a0 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:14:23.770 7f490d0224a0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:14:23.770 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:14:23.770 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2778 09/01/2025 02:14:23.770 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:14:23.771 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Connect Kube API result 0 09/01/2025 02:14:23.771 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - _schedule_vnc_sync 09/01/2025 02:14:23.771 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Start Watching request https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2778'})(timeout=120) 09/01/2025 02:14:23.773 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Service ns=None) 09/01/2025 02:14:23.773 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service start 09/01/2025 02:14:23.773 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Service done 09/01/2025 02:14:23.773 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:14:23.781 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Watches https://10.0.0.50:6443/api/v1/services ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2778'}) 09/01/2025 02:14:24.532 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:24.533 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:24.533 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:24.533 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:24.533 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:24.533 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:24.533 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:24.533 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:24.534 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.534 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:24.534 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.534 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.534 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:24.534 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.534 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:28.535 7f490e2baea0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:28.535 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:28.535 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:28.536 7f490e2baea0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.502 7f490d022680 [contrail-kube-manager] [ERROR]: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:14:31.503 7f490d022680 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError PodMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:14:31.503 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:14:31.503 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2826 09/01/2025 02:14:31.503 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:14:31.503 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Connect Kube API result 0 09/01/2025 02:14:31.504 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 02:14:31.504 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Start Watching request https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2826'})(timeout=120) 09/01/2025 02:14:31.505 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 02:14:31.505 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 02:14:31.505 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 02:14:31.505 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:14:31.520 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Watches https://10.0.0.50:6443/api/v1/pods ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2826'}) 09/01/2025 02:14:32.537 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:32.538 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:32.538 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:32.539 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:36.541 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:36.543 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:36.543 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:36.543 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:36.543 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:36.543 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:36.543 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:36.544 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:36.544 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:36.544 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:36.544 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:36.544 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:36.544 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:36.544 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:40.543 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:40.544 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:40.544 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:40.545 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:44.545 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:44.546 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:44.546 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:44.546 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:44.546 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:44.546 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:44.546 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:44.547 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:44.547 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:44.547 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:44.547 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:44.547 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:44.547 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:44.547 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:48.393 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=3453 newResVer=3453 09/01/2025 02:14:48.394 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:14:48.394 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:14:48.394 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:14:48.546 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:48.547 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:48.547 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:48.547 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:48.548 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.988 7f490d10c180 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:14:52.549 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:52.550 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:52.550 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:52.550 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:52.550 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:52.550 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:52.550 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:52.551 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:52.551 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:52.551 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:52.551 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:52.551 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:52.551 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:52.551 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.528 7f490d10c180 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:14:56.550 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:56.551 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:56.552 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:56.552 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:56.552 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:56.552 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:56.552 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:56.552 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:56.553 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:56.553 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:56.554 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:56.554 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:56.554 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:56.554 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:58.202 7f490d10c180 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:15:00.553 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:00.554 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:00.554 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:00.554 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:00.554 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:00.555 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:04.555 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:04.563 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:04.563 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:04.563 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:04.563 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:04.563 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:04.563 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:04.563 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:04.563 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:04.564 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:04.564 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:04.564 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:04.564 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:04.564 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:06.913 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=2774 newResVer=3596 09/01/2025 02:15:06.914 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:15:06.914 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:15:06.914 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:15:08.565 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:08.566 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:08.566 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:08.566 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:08.566 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:08.566 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:08.567 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:08.567 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:08.567 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:08.567 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:08.567 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:08.567 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:08.567 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:08.567 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:12.569 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:12.570 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:12.571 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:12.571 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:12.571 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:12.571 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:16.571 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:16.572 7f490d0ccae0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:16.572 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:16.572 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:16.572 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:16.573 7f490d0ccae0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:17.217 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=2776 newResVer=3599 09/01/2025 02:15:17.218 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:15:17.218 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:15:17.218 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:15:20.575 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:20.576 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:20.576 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:20.576 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:20.576 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:20.576 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:20.576 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:20.577 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:20.577 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:20.577 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:20.577 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:20.577 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:20.577 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:20.577 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:21.701 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=3568 newResVer=3568 09/01/2025 02:15:21.701 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:15:21.701 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:15:21.701 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:15:23.666 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=2778 newResVer=3601 09/01/2025 02:15:23.666 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:15:23.666 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:15:23.666 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:15:24.578 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:24.579 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:24.580 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:24.580 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:24.580 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:24.580 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:28.579 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:28.580 7f490d0794a0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:28.581 7f490d0794a0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.361 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=2826 newResVer=3664 09/01/2025 02:15:31.362 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:15:31.362 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:15:31.362 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:15:32.582 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:32.583 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:32.584 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:32.585 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:32.585 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:36.587 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:36.588 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:36.589 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:36.589 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:36.589 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:40.589 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:40.590 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:40.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:40.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:40.590 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:40.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:40.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:40.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:40.590 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:40.591 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:40.591 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:40.591 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:40.591 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:40.591 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:44.592 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:44.593 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:44.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:44.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:44.593 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:44.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:44.593 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:44.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:44.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:44.594 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:44.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:44.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:44.594 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:44.594 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:48.540 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=3453 newResVer=3453 09/01/2025 02:15:48.540 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:15:48.540 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:15:48.540 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:15:48.594 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:48.595 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:48.595 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:48.595 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:48.595 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:48.595 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:48.595 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:48.595 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:48.596 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:48.596 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:48.596 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:48.596 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:48.596 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:48.596 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:52.596 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:52.597 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:52.597 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:52.597 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:52.598 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:52.599 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:56.598 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:56.599 7f490d0ed360 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:56.600 7f490d0ed360 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:56.601 7f490d0ed360 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.981 7f490d022900 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:15:59.981 7f490d022900 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.50', port=6443): Read timed out. 09/01/2025 02:15:59.981 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:15:59.982 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Re-registering event handler. resource_version_valid=True, resource_version=2897 09/01/2025 02:15:59.982 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Try to connect Kube API 10.0.0.50:6443 09/01/2025 02:15:59.982 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Connect Kube API result 0 09/01/2025 02:15:59.982 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - _schedule_vnc_sync 09/01/2025 02:15:59.982 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Start Watching request https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2897'})(timeout=120) 09/01/2025 02:15:59.983 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkAttachmentDefinition ns=None) 09/01/2025 02:15:59.984 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkAttachmentDefinition - no handler - skip 09/01/2025 02:15:59.984 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:15:59.996 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Watches https://10.0.0.50:6443/apis/k8s.cni.cncf.io/v1/network-attachment-definitions ({'watch': 'true', 'allowWatchBookmarks': 'true', 'resourceVersion': '2897'}) 09/01/2025 02:16:00.601 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:00.602 7f490d0edcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:00.602 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:00.602 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:00.603 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:04.603 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:04.605 7f490d0edcc0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:04.606 7f490d0edcc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:04.606 7f490d0edcc0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:06.236 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=3596 newResVer=3596 09/01/2025 02:16:06.236 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:16:06.236 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:16:06.236 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:16:08.606 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:08.607 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:08.608 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:12.608 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:12.609 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:12.609 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:12.609 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:12.609 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:12.609 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:12.610 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:12.610 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:12.610 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:12.610 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:12.610 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:12.610 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:12.610 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:12.610 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:16.610 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:16.612 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:16.613 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:16.613 7f490d155d60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:16.613 7f490d155d60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:16.613 7f490d155d60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:17.897 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=3599 newResVer=3599 09/01/2025 02:16:17.897 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:16:17.897 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:16:17.897 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:16:20.612 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:20.613 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:20.614 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:20.615 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:22.234 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=3568 newResVer=3568 09/01/2025 02:16:22.234 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:16:22.234 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - _schedule_vnc_sync 09/01/2025 02:16:22.234 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:16:22.234 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=NetworkPolicy ns=None) 09/01/2025 02:16:22.235 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy start 09/01/2025 02:16:22.235 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:16:22.237 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - NetworkPolicy done 09/01/2025 02:16:22.237 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:16:24.043 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=3601 newResVer=3601 09/01/2025 02:16:24.044 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:16:24.044 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:16:24.044 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:16:24.614 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:24.615 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:24.616 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:24.617 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:28.616 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:28.617 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:28.618 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:28.618 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:28.618 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:28.618 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:28.618 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:28.618 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:28.618 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:28.619 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:28.619 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:28.619 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:28.619 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:28.619 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:31.906 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=3664 newResVer=3664 09/01/2025 02:16:31.907 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:16:31.907 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:16:31.907 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:16:32.619 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:32.620 7f490d0ae540 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:32.620 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:32.620 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:32.620 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:32.620 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:32.620 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:32.621 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:32.621 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:32.621 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:32.621 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:32.621 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:32.621 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:32.621 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:36.622 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:36.624 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:36.625 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:36.625 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:36.625 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:37.074 7f490d0ae540 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map 09/01/2025 02:16:40.624 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:40.625 7f490d0ae540 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:40.626 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:44.627 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:44.628 7f490d0ae540 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:44.628 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:44.628 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:44.628 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:44.628 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:44.628 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:44.629 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:44.629 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:44.629 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:44.629 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:44.629 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:44.629 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:44.629 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:48.631 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:48.632 7f490d0ae540 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:48.632 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:48.632 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:48.632 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:48.632 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:48.632 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:48.632 7f490d0ae540 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:48.632 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:48.633 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:48.633 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:48.633 7f490d0ae540 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:48.633 7f490d0ae540 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:48.633 7f490d0ae540 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:48.829 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=3453 newResVer=3453 09/01/2025 02:16:48.829 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:16:48.829 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - _schedule_vnc_sync 09/01/2025 02:16:48.829 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:16:48.829 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Ingress ns=None) 09/01/2025 02:16:48.829 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress start 09/01/2025 02:16:48.830 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Ingress done 09/01/2025 02:16:48.830 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:16:48.830 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:16:52.632 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:52.633 7f490d02e7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:52.634 7f490d02e7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:56.636 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:56.637 7f490d02e7c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:56.637 7f490d02e7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:56.637 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:56.637 7f490d02e7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:56.637 7f490d02e7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:56.637 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:56.638 7f490d02e7c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:56.638 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:56.638 7f490d02e7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:56.638 7f490d02e7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:56.638 7f490d02e7c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:56.638 7f490d02e7c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:56.638 7f490d02e7c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:00.473 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=2897 newResVer=3690 09/01/2025 02:17:00.473 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:17:00.473 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:17:00.473 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:17:00.638 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:00.639 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:00.639 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:00.639 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:00.640 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:04.640 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:04.641 7f490d0cc680 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:04.641 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:04.641 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:04.641 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:04.642 7f490d0cc680 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:06.452 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=3596 newResVer=3596 09/01/2025 02:17:06.452 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:17:06.453 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:17:06.453 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:17:08.642 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:08.643 7f490e2bae00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:08.643 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:08.643 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:08.643 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:08.643 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:08.643 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:08.643 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:08.644 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:08.644 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:08.644 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:08.644 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:08.644 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:08.644 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:12.645 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:12.646 7f490e2bae00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:12.646 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:12.646 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:12.646 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:12.646 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:12.646 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:12.647 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:12.647 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:12.647 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:12.647 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:12.647 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:12.647 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:12.647 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:16.646 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:16.647 7f490e2bae00 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:16.648 7f490e2bae00 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:16.649 7f490e2bae00 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:16.649 7f490e2bae00 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:17.790 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=3599 newResVer=3599 09/01/2025 02:17:17.790 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:17:17.790 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:17:17.790 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:17:20.650 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:20.670 7f490e2bad60 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:20.671 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:20.671 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:20.671 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:20.671 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:20.671 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:20.671 7f490e2bad60 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:20.671 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:20.672 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:20.672 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:20.672 7f490e2bad60 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:20.672 7f490e2bad60 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:20.672 7f490e2bad60 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:21.300 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=3568 newResVer=3568 09/01/2025 02:17:21.300 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:17:21.300 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:17:21.300 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:17:24.046 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=3601 newResVer=3601 09/01/2025 02:17:24.046 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:17:24.046 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:17:24.046 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:17:24.671 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:24.672 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:24.672 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:24.672 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:24.673 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:28.673 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:28.674 7f490d10c900 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:28.675 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:28.675 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:28.675 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:28.675 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:28.675 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:28.676 7f490d10c900 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:28.676 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:28.676 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:28.676 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:28.676 7f490d10c900 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:28.676 7f490d10c900 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:28.676 7f490d10c900 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:31.757 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=3664 newResVer=3664 09/01/2025 02:17:31.757 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:17:31.757 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:17:31.757 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:17:32.676 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:32.677 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:32.677 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:32.677 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:32.678 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:36.678 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:36.679 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:36.680 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:36.681 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:40.680 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:40.681 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:40.681 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:40.681 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:40.682 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:44.682 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:44.684 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:44.685 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:44.685 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:44.685 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:48.685 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:48.686 7f490d10c180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:48.686 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:48.686 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:48.686 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:48.686 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:48.686 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:48.687 7f490d10c180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:48.687 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:48.687 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:48.687 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:48.687 7f490d10c180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:48.687 7f490d10c180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:48.687 7f490d10c180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:49.097 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=3453 newResVer=3453 09/01/2025 02:17:49.097 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:17:49.097 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:17:49.097 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:17:52.686 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:52.687 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:52.688 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:56.689 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:56.690 7f490d10c220 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:56.690 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:56.690 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:56.690 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:56.690 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:56.690 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:56.691 7f490d10c220 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:56.691 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:56.691 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:56.691 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:56.691 7f490d10c220 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:56.691 7f490d10c220 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:56.691 7f490d10c220 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:00.389 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - Received BOOKMARK: oldResVer=3690 newResVer=3690 09/01/2025 02:18:00.389 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:18:00.389 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:18:00.389 7f490d022900 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkMonitor - start process kube event 09/01/2025 02:18:00.691 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:00.692 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:00.692 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:00.692 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:00.692 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:00.693 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:00.693 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:00.693 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:00.693 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:00.693 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:00.693 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:00.693 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:00.694 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:00.694 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:04.693 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:04.694 7f490d0792c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:04.695 7f490d0792c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:04.696 7f490d0792c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:04.696 7f490d0792c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:06.378 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - Received BOOKMARK: oldResVer=3596 newResVer=3596 09/01/2025 02:18:06.378 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:18:06.378 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - _schedule_vnc_sync 09/01/2025 02:18:06.378 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:18:06.379 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Namespace ns=None) 09/01/2025 02:18:06.379 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace start 09/01/2025 02:18:06.379 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Namespace done 09/01/2025 02:18:06.379 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:18:06.379 7f490d022a40 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NamespaceMonitor - start process kube event 09/01/2025 02:18:08.697 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:08.698 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:08.698 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:08.698 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:08.698 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:08.698 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:08.698 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:08.698 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:08.698 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:08.699 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:08.699 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:08.699 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:08.699 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:08.699 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:12.698 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:12.699 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:12.699 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:12.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:16.700 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:16.701 7f490d0ed180 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:16.701 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:16.701 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:16.701 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:16.701 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:16.701 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:16.702 7f490d0ed180 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:16.702 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:16.702 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:16.702 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:16.702 7f490d0ed180 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:16.702 7f490d0ed180 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:16.702 7f490d0ed180 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:18.055 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - Received BOOKMARK: oldResVer=3599 newResVer=3599 09/01/2025 02:18:18.055 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:18:18.055 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - _schedule_vnc_sync 09/01/2025 02:18:18.056 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:18:18.056 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Endpoints ns=None) 09/01/2025 02:18:18.056 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Endpoints - no handler - skip 09/01/2025 02:18:18.056 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:18:18.056 7f490d079ea0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: EndPointMonitor - start process kube event 09/01/2025 02:18:20.702 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:20.703 7f490d1552c0 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:20.703 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:20.704 7f490d1552c0 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:21.451 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - Received BOOKMARK: oldResVer=3568 newResVer=3568 09/01/2025 02:18:21.452 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:18:21.452 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:18:21.452 7f490d0222c0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: NetworkPolicyMonitor - start process kube event 09/01/2025 02:18:23.270 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - Received BOOKMARK: oldResVer=3601 newResVer=3601 09/01/2025 02:18:23.270 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:18:23.270 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:18:23.270 7f490d0224a0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: ServiceMonitor - start process kube event 09/01/2025 02:18:24.705 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:24.706 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:24.706 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:24.706 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:24.707 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:28.707 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:28.709 7f490d155a40 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:28.709 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:28.709 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:28.709 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:28.709 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:28.709 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:28.710 7f490d155a40 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:28.710 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:28.710 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:28.710 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:28.710 7f490d155a40 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:28.710 7f490d155a40 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:28.710 7f490d155a40 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:32.249 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - Received BOOKMARK: oldResVer=3664 newResVer=3664 09/01/2025 02:18:32.249 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:18:32.249 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - _schedule_vnc_sync 09/01/2025 02:18:32.249 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:18:32.249 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - Process event (name=None event_type=TF_VNC_SYNC kind=Pod ns=None) 09/01/2025 02:18:32.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod start 09/01/2025 02:18:32.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - vnc_sync - Pod done 09/01/2025 02:18:32.250 7f490d022ae0 [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncKubernetes - wait event (qsize=0 timeout=120) 09/01/2025 02:18:32.250 7f490d022680 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: PodMonitor - start process kube event 09/01/2025 02:18:32.710 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:32.711 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:32.712 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:32.712 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:32.712 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:36.714 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:36.715 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:36.715 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:36.715 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:36.715 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:36.715 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:36.715 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:36.715 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:36.715 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:36.716 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:36.716 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:36.716 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:36.716 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:36.716 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:40.716 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:40.717 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:40.717 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:40.717 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:40.717 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:40.717 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:40.717 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:40.718 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:40.718 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:40.718 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:40.718 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:40.718 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:40.718 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:40.718 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:44.719 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:44.720 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:44.720 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:44.720 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:44.720 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:44.720 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:44.720 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:44.720 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:44.721 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:44.721 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:44.721 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:44.721 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:44.721 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:44.721 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:48.721 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - Received BOOKMARK: oldResVer=3453 newResVer=3453 09/01/2025 02:18:48.721 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:48.722 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:18:48.722 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: empty line received 09/01/2025 02:18:48.722 7f490d079ae0 [contrail-kube-manager] [INFO]: __default__ [SYS_INFO]: KubeManagerInfoLog: IngressMonitor - start process kube event 09/01/2025 02:18:48.722 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:48.722 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:48.722 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:48.723 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:52.723 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:18:52.724 7f490d0ae040 [contrail-kube-manager] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:18:52.724 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:18:52.724 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:52.724 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:18:52.724 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:18:52.724 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:18:52.725 7f490d0ae040 [contrail-kube-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:18:52.725 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:52.725 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1 process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.50, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.50:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.50:8082, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.50:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:18:52.725 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:52.725 7f490d0ae040 [contrail-kube-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:52.725 7f490d0ae040 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-k8s-1454-1:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8108 start_time = 1756690494875696 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:18:52.725 7f490d0ae040 [contrail-kube-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:18:54.722 7f490d0ae040 [contrail-kube-manager] [DEBUG]: send sync_uve in the [ObjectKubernetesManagerNode:NodeStatusUVE] map