08/25/2025 01:45:41.441 7f07213fb048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/25/2025 01:45:41.442 7f07213fb048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 start_time = 1756086341441248 collector_name = collector_ip = collector_list = [ 10.0.0.48:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 08/25/2025 01:45:41.810 7f07241eac88 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.48:5673//', 'pyamqp://guest:guest@:5673//'] 08/25/2025 01:45:41.883 7f07241eac88 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 08/25/2025 01:45:41.884 7f07212ce630 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/25/2025 01:45:41.893 7f07212ce630 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/25/2025 01:45:41.903 7f07241eac88 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/25/2025 01:45:41.904 7f07212e5cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/25/2025 01:45:41.949 7f07241eac88 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 08/25/2025 01:45:41.950 7f07212ceb70 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 08/25/2025 01:45:41.962 7f07241eac88 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/25/2025 01:45:41.963 7f0729127048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/25/2025 01:45:42.002 7f07212e9630 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.48:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 08/25/2025 01:45:42.030 7f07241eac88 [contrail-kube-manager] [WARNING]: Waiting to be elected as master... 08/25/2025 01:45:42.030 7f07241eac88 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master... 08/25/2025 01:45:42.030 7f072004b438 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.48:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 08/25/2025 01:45:42.186 7f07241eac88 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing... 08/25/2025 01:45:42.186 7f07241eac88 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing... 08/25/2025 01:45:42.405 7f072128e048 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port 08/25/2025 01:45:42.405 7f072128e048 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108 08/25/2025 01:45:42.806 7f0720dc8ac8 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.48,:8082, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 08/25/2025 01:45:42.817 7f07241eac88 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:42.818 7f07241eac88 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:42.831 7f0720dc8ac8 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.48,:8082, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 08/25/2025 01:45:45.446 7f0720dc8ac8 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-239-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.48, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.48:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.48,:8082, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down >>, ] >> 08/25/2025 01:45:45.886 7f07241eac88 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:45.899 7f07241eac88 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:48.919 7f07241eac88 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:48.919 7f07241eac88 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:51.935 7f07241eac88 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:51.936 7f07241eac88 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:54.953 7f07241eac88 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:54.953 7f07241eac88 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:57.976 7f07241eac88 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 08/25/2025 01:45:57.976 7f07241eac88 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 08/25/2025 01:46:00.999 7f07241eac88 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 08/25/2025 01:46:00.999 7f07241eac88 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 08/25/2025 01:46:04.058 7f07241eac88 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 08/25/2025 01:46:04.059 7f07241eac88 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 08/25/2025 01:46:07.079 7f07241eac88 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql 08/25/2025 01:46:07.080 7f07241eac88 [contrail-kube-manager] [WARNING]: initializing cassandra cluster 08/25/2025 01:46:07.080 7f07241eac88 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 08/25/2025 01:46:07.082 7f07241eac88 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 08/25/2025 01:46:07.193 7f07241eac88 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 08/25/2025 01:46:07.206 7f07241eac88 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 08/25/2025 01:46:07.220 7f07241eac88 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 08/25/2025 01:46:07.295 7f07241eac88 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 08/25/2025 01:46:07.303 7f07241eac88 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.48:5673//', 'pyamqp://guest:guest@:5673//'] 08/25/2025 01:46:07.303 7f07241eac88 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 08/25/2025 01:46:07.326 7f07241eac88 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/25/2025 01:46:07.344 7f07241eac88 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 08/25/2025 01:46:07.366 7f07241eac88 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/25/2025 01:46:07.376 7f071ffb8d68 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 08/25/2025 01:46:07.432 7f071ffb8d68 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 08/25/2025 01:46:07.568 7f071ffb8eb8 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 08/25/2025 01:46:07.568 7f071ffb8eb8 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 08/25/2025 01:46:07.603 7f071ffb8eb8 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/25/2025 01:46:07.804 7f071ffb8eb8 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 08/25/2025 01:46:07.804 7f071ffb8eb8 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 08/25/2025 01:46:07.853 7f071ffb8eb8 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 08/25/2025 01:48:29.487 7f071fe6d048 [contrail-kube-manager] [ERROR]: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.489 7f071fe6d048 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError IngressMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.501 7f071fe6db70 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.502 7f071fe6db70 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.548 7f071ddd78d0 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.549 7f071ddd78d0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.561 7f071dd84c18 [contrail-kube-manager] [ERROR]: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.561 7f071dd84c18 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError ServiceMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.561 7f071ddd8048 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:48:29.561 7f071ddd8048 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:49:48.275 7f071ffb8eb8 [contrail-kube-manager] [WARNING]: Object virtual_machine uuid 9bf56c28-b1f6-45da-9017-298e1672e102 was not found for operation DELETE 08/25/2025 01:49:48.275 7f071ffb8eb8 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object virtual_machine uuid 9bf56c28-b1f6-45da-9017-298e1672e102 was not found for operation DELETE 08/25/2025 01:50:29.584 7f071fe6db70 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:50:29.584 7f071fe6db70 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:50:29.598 7f071ddd78d0 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:50:29.598 7f071ddd78d0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:52:29.618 7f071ddd78d0 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:52:29.618 7f071ddd78d0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:54:22.717 7f071ffb8eb8 [contrail-kube-manager] [WARNING]: Object virtual_machine uuid aa91a29e-34d7-4906-9fdf-e8c9a476024c was not found for operation DELETE 08/25/2025 01:54:22.718 7f071ffb8eb8 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object virtual_machine uuid aa91a29e-34d7-4906-9fdf-e8c9a476024c was not found for operation DELETE 08/25/2025 01:54:29.633 7f071ddd78d0 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:54:29.633 7f071ddd78d0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:54:42.212 7f071dda1588 [contrail-kube-manager] [WARNING]: Delete endpoints event received while service ctest-nginx-svc-84411727 does not exist 08/25/2025 01:54:42.212 7f071dda1588 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Delete endpoints event received while service ctest-nginx-svc-84411727 does not exist 08/25/2025 01:56:29.655 7f071ddd78d0 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out. 08/25/2025 01:56:29.659 7f071ddd78d0 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.48', port=6443): Read timed out.