11/07/2024 02:02:51.033 7ff29d4b1828 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 11/07/2024 02:02:51.033 7ff29d4b1828 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 11/07/2024 02:02:51.034 7ff29d4b7198 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:02:51.034 7ff29d4b7198 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1730944971033998 collector_name = collector_ip = collector_list = [ 10.0.0.29:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 11/07/2024 02:02:51.035 7ff2b19226d8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.29:6379[Redis Instance initialized] connection down >>, ] >> 11/07/2024 02:02:51.035 7ff2b19226d8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.29:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 11/07/2024 02:02:51.037 7ff2b19798d0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 11/07/2024 02:02:51.037 7ff2b19798d0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.29:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 11/07/2024 02:02:51.171 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.29:5673//'] 11/07/2024 02:02:51.194 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 11/07/2024 02:02:51.195 7ff29cb4acc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.29:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:02:51.196 7ff29cb4acc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.29:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:02:51.197 7ff29cb4af60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:02:51.210 7ff29cb4acc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper re-established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:02:51.210 7ff29cb4acc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:02:51.223 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 11/07/2024 02:02:51.224 7ff29cb7f198 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:02:51.272 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:02:51.273 7ff29cb7fcc0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 11/07/2024 02:02:51.314 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:02:51.318 7ff29cb7feb8 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 11/07/2024 02:02:51.319 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 11/07/2024 02:02:51.319 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 11/07/2024 02:02:51.319 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 11/07/2024 02:02:51.320 7ff29cb4a780 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 11/07/2024 02:02:51.320 7ff29cb7fd68 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.29:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:02:51.320 7ff29cb7fd68 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.29:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:02:51.327 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 11/07/2024 02:02:51.352 7ff29cb4a780 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 11/07/2024 02:02:51.352 7ff29cbd3048 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.29:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:02:51.465 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 11/07/2024 02:02:51.466 7ff29cafa978 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.29:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 11/07/2024 02:02:51.520 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 11/07/2024 02:02:51.520 7ff29cbd9588 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.29:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:02:51.634 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 11/07/2024 02:02:51.885 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 11/07/2024 02:02:52.050 7ff29cacc198 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.29:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Up description = >>, << type = Redis-UVE name = AggregateRedis server_addrs = [ 10.0.0.29:6379, ] status = Up description = Connected to Aggregate Redis >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:02:52.681 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:02:53.157 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: table 'obj_fq_name_table', created 11/07/2024 02:02:53.177 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 11/07/2024 02:02:53.992 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:02:54.270 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 11/07/2024 02:02:54.277 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 11/07/2024 02:02:55.036 7ff29cad38d0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-k8s-manifests-107-1.cluster.local process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.29:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.29:6379 server_addrs = [ 10.0.0.29:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.29:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.29:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.29:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.29:2181, ] status = Up description = >>, << type = Redis-UVE name = AggregateRedis server_addrs = [ 10.0.0.29:6379, ] status = Up description = Connected to Aggregate Redis >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 11/07/2024 02:02:55.511 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:02:55.529 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: table 'obj_shared_table', created 11/07/2024 02:02:55.561 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 11/07/2024 02:02:56.360 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Zookeeper lock acquired. Performing schema change... 11/07/2024 02:02:56.837 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 11/07/2024 02:02:56.848 7ff29d4b7c18 [contrail-alarm-gen] [WARNING]: Zookeeper lock released. 11/07/2024 02:02:56.892 7ff29d4b7c18 [contrail-alarm-gen] [ERROR]: Error in db walk read 'type' 11/07/2024 02:03:11.441 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:0 11/07/2024 02:03:11.441 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:1 11/07/2024 02:03:11.441 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:2 11/07/2024 02:03:11.441 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:3 11/07/2024 02:03:11.441 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:4 11/07/2024 02:03:11.441 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:5 11/07/2024 02:03:11.441 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:6 11/07/2024 02:03:11.441 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:7 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:8 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:9 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:10 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:11 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:12 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:13 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:14 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:15 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:16 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:17 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:18 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:19 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:20 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:21 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:22 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:23 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:24 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:25 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:26 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:27 11/07/2024 02:03:11.442 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:28 11/07/2024 02:03:11.443 7ff29d4580f0 [contrail-alarm-gen] [ERROR]: Already a greenlet running to acquire:29