06/23/2025 01:35:30.384 7fa4c5497048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/23/2025 01:35:30.385 7fa4c5497048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local:KubernetesMgr:contrail-kube-manager:0 client_info = << status = Idle successful_connections = 0 pid = 1 start_time = 1750642530384663 collector_name = collector_ip = collector_list = [ 10.0.0.65:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 06/23/2025 01:35:30.483 7fa4c82c0808 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.65:5673//', 'pyamqp://guest:guest@:5673//'] 06/23/2025 01:35:30.498 7fa4c82c0808 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:30.499 7fa4c523c588 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 06/23/2025 01:35:30.499 7fa4c523c588 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 06/23/2025 01:35:30.520 7fa4c82c0808 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:30.521 7fa4c5243cc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/23/2025 01:35:30.552 7fa4c82c0808 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 06/23/2025 01:35:30.552 7fa4c523cb70 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 06/23/2025 01:35:30.564 7fa4c82c0808 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 06/23/2025 01:35:30.564 7fa4cd1e3048 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/23/2025 01:35:30.606 7fa4c52452e8 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.65:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:Zookeeper[] connection down >>, ] >> 06/23/2025 01:35:30.611 7fa4c82c0808 [contrail-kube-manager] [WARNING]: Waiting to be elected as master... 06/23/2025 01:35:30.611 7fa4c82c0808 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Waiting to be elected as master... 06/23/2025 01:35:30.612 7fa4c5200438 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.65:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 06/23/2025 01:35:30.781 7fa4c82c0808 [contrail-kube-manager] [WARNING]: Elected master kube-manager node. Initializing... 06/23/2025 01:35:30.781 7fa4c82c0808 [contrail-kube-manager] [WARNING]: __default__ [SYS_NOTICE]: KubeManagerNoticeLog: Elected master kube-manager node. Initializing... 06/23/2025 01:35:30.819 7fa4c521af60 [contrail-kube-manager] [ERROR]: Cannot write http_port 8108 to /tmp/contrail-kube-manager.0.http_port 06/23/2025 01:35:30.819 7fa4c521af60 [contrail-kube-manager] [ERROR]: Starting Introspect on HTTP Port 8108 06/23/2025 01:35:30.951 7fa4c496ccc0 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.65:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.65,:8082, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 06/23/2025 01:35:30.956 7fa4c82c0808 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:30.957 7fa4c82c0808 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:30.957 7fa4c494dd68 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.65:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.65,:8082, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], ApiServer:ApiServer[] connection down >>, ] >> 06/23/2025 01:35:33.977 7fa4c82c0808 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:33.977 7fa4c82c0808 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:34.409 7fa4c34f5588 [contrail-kube-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-221-1.cluster.local process_status = [ << module_id = contrail-kube-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.65:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.65, , ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.65:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.65,:8082, ] status = Down description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[] connection down >>, ] >> 06/23/2025 01:35:37.006 7fa4c82c0808 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:37.006 7fa4c82c0808 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:40.035 7fa4c82c0808 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:40.036 7fa4c82c0808 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:43.071 7fa4c82c0808 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:43.071 7fa4c82c0808 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:46.110 7fa4c82c0808 [contrail-kube-manager] [ERROR]: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:46.111 7fa4c82c0808 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: VncKubernetes - vnc_connect failed: 06/23/2025 01:35:49.191 7fa4c82c0808 [contrail-kube-manager] [WARNING]: VNCCassandra started with driver cql 06/23/2025 01:35:49.192 7fa4c82c0808 [contrail-kube-manager] [WARNING]: initializing cassandra cluster 06/23/2025 01:35:49.192 7fa4c82c0808 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 06/23/2025 01:35:49.194 7fa4c82c0808 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 06/23/2025 01:35:49.811 7fa4c82c0808 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 06/23/2025 01:35:49.853 7fa4c82c0808 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 06/23/2025 01:35:50.257 7fa4c82c0808 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 06/23/2025 01:35:50.471 7fa4c82c0808 [contrail-kube-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 06/23/2025 01:35:50.509 7fa4c82c0808 [contrail-kube-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.65:5673//', 'pyamqp://guest:guest@:5673//'] 06/23/2025 01:35:50.509 7fa4c82c0808 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:50.546 7fa4c82c0808 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:50.565 7fa4c82c0808 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 06/23/2025 01:35:50.584 7fa4c82c0808 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 06/23/2025 01:35:50.589 7fa4c4972828 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection down 06/23/2025 01:35:50.631 7fa4c4972828 [contrail-kube-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 06/23/2025 01:35:50.727 7fa4c33e9c18 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 06/23/2025 01:35:50.727 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:50.764 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:51.058 7fa4c33e9c18 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 06/23/2025 01:35:51.058 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:51.095 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:51.122 7fa4c33e9c18 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 06/23/2025 01:35:51.122 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:51.170 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:51.199 7fa4c33e9c18 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 06/23/2025 01:35:51.199 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:51.234 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:51.291 7fa4c33e9c18 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 06/23/2025 01:35:51.294 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:51.332 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:51.432 7fa4c33e9c18 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 06/23/2025 01:35:51.433 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:51.517 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:51.688 7fa4c33e9c18 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 06/23/2025 01:35:51.688 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:51.717 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:35:51.752 7fa4c33e9c18 [contrail-kube-manager] [ERROR]: Error in rabbitmq drainer greenlet: 'KubeManagerLogger' object has no attribute 'warn' 06/23/2025 01:35:51.752 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection down 06/23/2025 01:35:51.789 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 06/23/2025 01:38:07.977 7fa4c42a1198 [contrail-kube-manager] [ERROR]: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:38:07.978 7fa4c42a1198 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkPolicyMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:38:08.011 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:38:08.012 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:38:08.028 7fa4c3b282e8 [contrail-kube-manager] [ERROR]: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:38:08.028 7fa4c3b282e8 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NamespaceMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:40:08.022 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:40:08.022 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:42:08.035 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:42:08.036 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:43:07.431 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: Object virtual_machine uuid 04280ca9-0327-4f2e-88da-b495d6caf4b1 was not found for operation DELETE 06/23/2025 01:43:07.431 7fa4c33e9c18 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Object virtual_machine uuid 04280ca9-0327-4f2e-88da-b495d6caf4b1 was not found for operation DELETE 06/23/2025 01:43:48.896 7fa4c3e79198 [contrail-kube-manager] [WARNING]: Delete endpoints event received while service ctest-nginx-svc-62391198 does not exist 06/23/2025 01:43:48.896 7fa4c3e79198 [contrail-kube-manager] [WARNING]: __default__ [SYS_WARN]: KubeManagerWarningLog: Delete endpoints event received while service ctest-nginx-svc-62391198 does not exist 06/23/2025 01:44:08.047 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:44:08.048 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:46:08.060 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out. 06/23/2025 01:46:08.060 7fa4c3b98a20 [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: IOError NetworkMonitor - HTTPSConnectionPool(host='10.0.0.65', port=6443): Read timed out.