09/14/2026 01:48:45.083 7f24f7a81360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 01:48:45.083 7f24f7a81360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 01:48:45.084 7f24f7a81860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 01:48:45.085 7f24f7a81860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789350525084555 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 01:48:45.086 7f24f7a81860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 01:48:45.086 7f24f7a81860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:48:45.117 7f24f7a30d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 01:48:45.118 7f24f7a30d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:48:45.390 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 01:48:45.416 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 01:48:45.418 7f24f79ec360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:48:45.419 7f24f79ec360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:48:45.420 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:48:45.429 7f24fa5d9540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:48:45.447 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 01:48:45.447 7f24fa5d9540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:48:45.524 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:48:45.526 7f24f79ece00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:48:45.529 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:48:45.547 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:48:45.547 7f24f79ece00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:48:45.550 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 01:48:45.550 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 01:48:45.550 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 01:48:45.550 7f24f79ece00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:48:45.551 7f24f7351400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:48:45.551 7f24f7351400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:48:45.553 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 01:48:45.582 7f24f79ece00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:48:45.583 7f24faeb60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:48:45.689 7f24f7a819a0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 01:48:45.691 7f24faeb6400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:48:45.826 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:48:46.247 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:48:46.822 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:48:47.583 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:48:49.087 7f24faeb6400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:48:49.346 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:48:52.027 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:48:58.046 7f24fa5d9900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:00.158 7f24fa5d9900 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 01:49:00.739 7f24f7a819a0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 01:49:00.739 7f24fc8838c0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 01:49:00.740 7f24fc8838c0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 01:49:10.986 7ff3058b3360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 01:49:10.986 7ff3058b3360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 01:49:10.987 7ff3058b3860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 01:49:10.987 7ff3058b3860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789350550986890 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 01:49:10.988 7ff3058b3860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 01:49:10.988 7ff3058b3860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:49:10.990 7ff305861c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 01:49:10.990 7ff305861c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:49:11.055 7ff3058b3860 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 01:49:11.067 7ff3058b3860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 01:49:11.068 7ff305820360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:49:11.068 7ff305820360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:49:11.069 7ff30840a860 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:11.070 7ff305861d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:49:11.079 7ff3058b3860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 01:49:11.079 7ff305861d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:49:11.102 7ff3058b3860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:49:11.103 7ff305820e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:49:11.113 7ff3058b3860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:49:11.114 7ff305820e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:49:11.114 7ff3058b3860 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 01:49:11.115 7ff3058b3860 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 01:49:11.115 7ff3058b3860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 01:49:11.115 7ff305820e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:49:11.116 7ff30574c400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:49:11.116 7ff30574c400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:49:11.118 7ff3058b3860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 01:49:11.134 7ff305820e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:49:11.135 7ff30579e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:49:11.153 7ff30840a860 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:11.192 7ff3058b3860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 01:49:11.194 7ff30579e400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:49:11.346 7ff30840a860 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:11.748 7ff30840a860 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:12.421 7ff30840a860 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:14.069 7ff30840a860 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:14.994 7ff30579e400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:49:17.973 7ff30840a860 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:25.089 7ff30840a860 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:25.994 7ff30840a860 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 01:49:26.265 7ff3058b3860 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 01:49:26.266 7ff30aba5600 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 01:49:26.266 7ff30aba5600 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 01:49:46.895 7fbc4b5ee360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 01:49:46.896 7fbc4b5ee360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 01:49:46.896 7fbc4b5ee860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 01:49:46.897 7fbc4b5ee860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789350586896797 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 01:49:46.897 7fbc4b5ee860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 01:49:46.898 7fbc4b5ee860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:49:46.899 7fbc4b59dc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 01:49:46.900 7fbc4b59dc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:49:46.968 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 01:49:46.979 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 01:49:46.980 7fbc4c452360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:49:46.980 7fbc4c452360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:49:46.982 7fbc4e146900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:46.983 7fbc4b59dd60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:49:46.988 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 01:49:46.989 7fbc4b59dd60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:49:47.001 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:49:47.002 7fbc4c452e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:49:47.006 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:49:47.007 7fbc4c452e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:49:47.007 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 01:49:47.007 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 01:49:47.007 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 01:49:47.008 7fbc4c452e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:49:47.008 7fbc4df98400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:49:47.009 7fbc4df98400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:49:47.012 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 01:49:47.019 7fbc4c452e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:49:47.019 7fbc4df98f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:49:47.042 7fbc4b5ee860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 01:49:47.049 7fbc4df98f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:49:47.074 7fbc4e146900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:47.215 7fbc4e146900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:47.591 7fbc4e146900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:48.123 7fbc4e146900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:48.995 7fbc4e146900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:50.900 7fbc4df98f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:49:51.142 7fbc4e146900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:49:56.886 7fbc4e146900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:01.910 7fbc4e146900 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 01:50:02.140 7fbc4b5ee860 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 01:50:02.141 7fbc501a5900 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 01:50:02.141 7fbc501a5900 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 01:50:35.763 7ffb3ab76360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 01:50:35.763 7ffb3ab76360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 01:50:35.764 7ffb3ab76860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 01:50:35.765 7ffb3ab76860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789350635764649 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 01:50:35.765 7ffb3ab76860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 01:50:35.765 7ffb3ab76860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:50:35.767 7ffb3ab25d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 01:50:35.767 7ffb3ab25d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:50:35.844 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 01:50:35.862 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 01:50:35.863 7ffb3b9eb4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:50:35.863 7ffb3b9eb4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:50:35.864 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:35.873 7ffb3d6ce540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:50:35.896 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 01:50:35.896 7ffb3d6ce540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:50:35.918 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:50:35.919 7ffb3b9ebf40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:50:35.923 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:50:35.923 7ffb3b9ebf40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:50:35.924 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 01:50:35.924 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 01:50:35.925 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 01:50:35.925 7ffb3b9ebf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:50:35.925 7ffb3aa81540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:50:35.926 7ffb3aa81540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:50:35.931 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 01:50:35.948 7ffb3b9ebf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:50:35.949 7ffb3a9eb220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:50:35.976 7ffb3ab769a0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 01:50:35.978 7ffb3a9eb540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:50:36.003 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:36.279 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:36.943 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:38.043 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:39.382 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:39.766 7ffb3a9eb540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:50:41.098 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:44.620 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:50.731 7ffb3d6ce900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:50:50.832 7ffb3d6ce900 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 01:50:51.079 7ffb3ab769a0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 01:50:51.079 7ffb3f83bf80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 01:50:51.079 7ffb3f83bf80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 01:51:52.873 7f439034f360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 01:51:52.873 7f439034f360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 01:51:52.874 7f439034f860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 01:51:52.874 7f439034f860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789350712874116 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 01:51:52.875 7f439034f860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 01:51:52.875 7f439034f860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:51:52.877 7f43902fdd60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 01:51:52.877 7f43902fdd60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:51:52.930 7f439034f860 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 01:51:52.938 7f439034f860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 01:51:52.939 7f439372b360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:51:52.939 7f439372b360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:51:52.940 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:51:52.941 7f4392ea6540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:51:52.942 7f439034f860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 01:51:52.943 7f4392ea6540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:51:52.953 7f439034f860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:51:52.953 7f439372be00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:51:52.956 7f439034f860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:51:52.956 7f439372be00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:51:52.957 7f439034f860 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 01:51:52.957 7f439034f860 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 01:51:52.957 7f439034f860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 01:51:52.957 7f439372be00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:51:52.958 7f43902d9400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:51:52.958 7f43902d9400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:51:52.960 7f439034f860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 01:51:52.965 7f439372be00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:51:52.965 7f438fc1e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:51:52.986 7f439034f860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 01:51:52.989 7f438fc1e400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:51:53.008 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:51:53.118 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:51:53.363 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:51:54.036 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:51:55.130 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:51:56.876 7f438fc1e400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:51:57.550 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:52:01.787 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:52:07.013 7f4392ea6900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:52:07.918 7f4392ea6900 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 01:52:07.997 7f439034f860 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 01:52:07.997 7f4394f0b640 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 01:52:07.997 7f4394f0b640 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 01:53:37.865 7fc910981360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 01:53:37.865 7fc910981360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 01:53:37.866 7fc910981860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 01:53:37.866 7fc910981860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789350817866014 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 01:53:37.867 7fc910981860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 01:53:37.867 7fc910981860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:53:37.868 7fc910930c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 01:53:37.869 7fc910930c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:53:37.951 7fc910981860 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 01:53:37.959 7fc910981860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 01:53:37.960 7fc9109f1360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:53:37.960 7fc9109f1360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:53:37.961 7fc913499900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:53:37.963 7fc910930d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:53:37.965 7fc910981860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 01:53:37.965 7fc910930d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:53:37.976 7fc910981860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:53:37.977 7fc9109f1e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:53:37.979 7fc910981860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:53:37.980 7fc9109f1e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:53:37.980 7fc910981860 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 01:53:37.980 7fc910981860 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 01:53:37.980 7fc910981860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 01:53:37.981 7fc9109f1e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:53:37.981 7fc910806400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:53:37.981 7fc910806400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:53:37.984 7fc910981860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 01:53:37.987 7fc9109f1e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:53:37.987 7fc910806cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:53:38.006 7fc910981860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 01:53:38.008 7fc913df1040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:53:38.044 7fc913499900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:53:38.260 7fc913499900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:53:38.700 7fc913499900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:53:39.440 7fc913499900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:53:40.369 7fc913499900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:53:41.867 7fc913df1040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:53:42.670 7fc913499900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:53:47.803 7fc913499900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:53:52.926 7fc913499900 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 01:53:53.062 7fc910981860 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 01:53:53.062 7fc9155329c0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 01:53:53.062 7fc9155329c0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 01:56:42.816 7f136adeb360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 01:56:42.816 7f136adeb360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 01:56:42.817 7f136adeb860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 01:56:42.817 7f136adeb860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789351002817407 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 01:56:42.818 7f136adeb860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 01:56:42.818 7f136adeb860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:56:42.820 7f136ad99d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 01:56:42.820 7f136ad99d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:56:42.872 7f136adeb860 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 01:56:42.879 7f136adeb860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 01:56:42.880 7f136ad4d360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:56:42.881 7f136ad4d360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:56:42.882 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:42.883 7f136d942540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:56:42.884 7f136adeb860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 01:56:42.885 7f136d942540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:56:42.897 7f136adeb860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:56:42.897 7f136ad4de00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 01:56:42.900 7f136adeb860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:56:42.901 7f136ad4de00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 01:56:42.901 7f136adeb860 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 01:56:42.901 7f136adeb860 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 01:56:42.901 7f136adeb860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 01:56:42.902 7f136ad4de00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 01:56:42.902 7f136acf8400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:56:42.902 7f136acf8400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:56:42.905 7f136adeb860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 01:56:42.910 7f136ad4de00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 01:56:42.910 7f136acf8f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 01:56:42.926 7f136adeb860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 01:56:42.928 7f136acf8f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:56:42.952 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:43.053 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:43.260 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:43.692 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:44.401 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:46.261 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:46.819 7f136acf8f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 01:56:48.701 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:52.550 7f136d942900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 01:56:57.910 7f136d942900 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 01:56:57.946 7f136adeb860 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 01:56:57.947 7f136f99da00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 01:56:57.947 7f136f99da00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:02:19.089 7f8a83aaa360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:02:19.089 7f8a83aaa360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:02:19.090 7f8a83aaa860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:02:19.090 7f8a83aaa860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789351339090123 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:02:19.091 7f8a83aaa860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:02:19.091 7f8a83aaa860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:02:19.092 7f8a83a58c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:02:19.093 7f8a83a58c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:02:19.145 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 02:02:19.152 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:02:19.153 7f8a85412360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:02:19.154 7f8a85412360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:02:19.155 7f8a86601900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:02:19.156 7f8a83a58d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:02:19.157 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:02:19.157 7f8a83a58d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:02:19.166 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:02:19.166 7f8a85412e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:02:19.169 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:02:19.169 7f8a85412e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:02:19.169 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:02:19.169 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:02:19.170 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:02:19.170 7f8a85412e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:02:19.170 7f8a839b4400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:02:19.170 7f8a839b4400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:02:19.173 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:02:19.176 7f8a85412e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:02:19.177 7f8a839b4f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:02:19.192 7f8a83aaa860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:02:19.193 7f8a839b4f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:02:19.246 7f8a86601900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:02:19.400 7f8a86601900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:02:19.601 7f8a86601900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:02:20.032 7f8a86601900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:02:20.960 7f8a86601900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:02:22.978 7f8a86601900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:02:23.091 7f8a839b4f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:02:26.225 7f8a86601900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:02:34.106 7f8a86601900 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:02:34.276 7f8a83aaa860 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:02:34.277 7f8a88664840 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:02:34.277 7f8a88664840 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:07:46.942 7fe0ec7b5360 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:07:46.942 7fe0ec7b5360 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:07:46.943 7fe0ec7b5860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:07:46.944 7fe0ec7b5860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789351666943779 collector_name = collector_ip = collector_list = [ 10.0.0.162:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:07:46.944 7fe0ec7b5860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:07:46.945 7fe0ec7b5860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:07:46.946 7fe0ec764cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:07:46.947 7fe0ec764cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:07:47.009 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.162:5673//'] 09/14/2026 02:07:47.017 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:07:47.018 7fe0ec73c360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:07:47.018 7fe0ec73c360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.162:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:07:47.019 7fe0ef30d900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:07:47.020 7fe0ec764f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:07:47.022 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:07:47.022 7fe0ec764f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:07:47.032 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:07:47.032 7fe0ec73ce00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:07:47.035 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:07:47.036 7fe0ec73ce00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:07:47.036 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:07:47.036 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:07:47.036 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:07:47.036 7fe0ec73ce00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:07:47.037 7fe0ec6c0400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:07:47.037 7fe0ec6c0400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:07:47.040 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:07:47.043 7fe0ec73ce00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:07:47.043 7fe0ec6c0cc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:07:47.058 7fe0ec7b5860 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:07:47.060 7fe0efc14040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:07:47.156 7fe0ef30d900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:07:47.496 7fe0ef30d900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:07:47.950 7fe0ef30d900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:07:49.065 7fe0ef30d900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:07:50.945 7fe0efc14040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-431-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.162:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.162:6379 server_addrs = [ 10.0.0.162:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.162:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.162:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.162:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:07:51.563 7fe0ef30d900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:07:55.521 7fe0ef30d900 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:08:01.980 7fe0ef30d900 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:08:02.083 7fe0ec7b5860 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:08:02.084 7fe0f136cac0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:08:02.084 7fe0f136cac0 [contrail-alarm-gen] [ERROR]: Stopped http server