09/14/2026 02:30:42.685 7f1877520700 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:30:42.685 7f1877520700 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:30:42.685 7f1877520700 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:30:42.694 7f1877520700 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:30:42.694 7f1872a6a2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:30:42.695 7f1872a6a2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:30:42.695 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:30:42.696 7f1872a6a7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:30:42.696 7f1872a6a7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:30:42.696 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:30:42.696 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.696 7f1872a6a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:30:42.697 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.697 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.697 7f1872a6a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353042696378 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.50:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:30:42.697 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.698 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.699 7f1872a6a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:30:42.699 7f1872a6a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.699 7f1872a1a2c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:30:42.699 7f1872a1a2c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:30:42.700 7f1872a6ae00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:30:42.701 7f1872a1a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.701 7f1872a1a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:30:42.702 7f1872a1a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.702 7f1872a1a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.702 7f1872a1a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:30:42.702 7f1872a1a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.776 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:30:42.785 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:30:42.786 7f1872977040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.787 7f1872977040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:30:42.787 7f1872977040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.787 7f1872977040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.787 7f1872977040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:30:42.787 7f1872977040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.788 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:42.789 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:42.789 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:42.791 7f1872a1ab80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.791 7f1872a1ab80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:30:42.791 7f1872a1ab80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.792 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:30:42.792 7f1872a1ab80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.792 7f1872a1ab80 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:30:42.792 7f1872a1ab80 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.793 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:42.793 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:42.801 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:42.802 7f1872977c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.802 7f1872977c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:30:42.802 7f1872977c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.804 7f1872977c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.804 7f1872977c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:30:42.804 7f1872977c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.827 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:30:42.828 7f1872977ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.828 7f1872977ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:30:42.828 7f1872977ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.832 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:30:42.832 7f1872977ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.832 7f1872977ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:30:42.833 7f1872977ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.833 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:30:42.833 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:30:42.834 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:30:42.834 7f1872977e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:30:42.835 7f18728e40e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.835 7f18728e40e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:30:42.835 7f18728e40e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.835 7f18728e40e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.836 7f18728e40e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:30:42.836 7f18728e40e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.840 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:30:42.841 7f1872977e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:30:42.842 7f18728e4860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.842 7f18728e4860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:30:42.842 7f18728e4860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.871 7f1872a6a7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:30:42.880 7f1874100400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.881 7f1874100400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:30:42.881 7f1874100400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:30:42.934 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:42.934 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:42.936 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:42.936 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:42.937 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:42.937 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:43.100 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:43.101 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:43.101 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:43.101 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:43.101 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:43.102 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:43.443 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:43.444 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:43.444 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:43.444 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:43.444 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:43.445 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:43.700 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:44.020 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:44.021 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:44.021 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:44.022 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:44.022 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:44.022 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:44.701 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:44.847 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:44.848 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:44.848 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:44.849 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:44.849 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:44.849 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:45.703 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:46.218 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:46.218 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:46.219 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:46.219 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:46.219 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:46.220 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:46.696 7f1874100400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:30:46.697 7f1874100400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:30:46.698 7f1874100400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:30:46.698 7f1874100400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:30:46.698 7f1874100400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:30:46.698 7f1874100400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:30:46.698 7f1874100400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:30:46.698 7f1874100400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:30:46.699 7f1874100400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:30:46.700 7f1874100400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:30:46.700 7f1874100400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:30:46.700 7f1874100900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:30:46.701 7f1874100900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:30:46.703 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:47.705 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:48.164 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:48.165 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:48.165 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:48.166 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:48.166 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:48.166 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:48.706 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:49.708 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:50.708 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:51.244 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:51.244 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:51.245 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:51.245 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:51.245 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:51.246 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:51.710 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:52.712 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:53.713 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:54.714 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:55.240 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:30:55.241 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:55.241 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:30:55.242 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:55.242 7f1872a1aea0 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:30:55.242 7f1872a1aea0 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:30:55.717 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:56.718 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:57.719 7f1872a1a400 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:30:57.762 7f1872a1aea0 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:30:57.762 7f1872a1aea0 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:30:57.763 7f18755c34a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:30:57.950 7f1872a6a7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:30:57.951 7f1877520700 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:30:57.951 7f1877520700 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:30:57.951 7f1872a1ac20 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:30:57.951 7f1872a1ac20 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:30:57.966 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:30:57.966 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:30:57.966 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:30:57.967 7f1877520700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:31:07.316 7f31b6128c00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:31:07.316 7f31b6128c00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:31:07.316 7f31b6128c00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:31:07.319 7f31b6128c00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:31:07.320 7f31b135b2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:31:07.320 7f31b135b2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:31:07.320 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:31:07.321 7f31b135b7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:31:07.321 7f31b135b7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:31:07.322 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:31:07.322 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.322 7f31b135b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:31:07.322 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.322 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.322 7f31b135b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353067321698 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.189:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:31:07.322 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.323 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.324 7f31b135b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:07.324 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.324 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.324 7f31b135b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:07.324 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.324 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.324 7f31b135b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:07.325 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.325 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.325 7f31b135b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:07.325 7f31b135b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.326 7f31b135be00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:31:07.326 7f31b135be00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:31:07.327 7f31b135bc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:31:07.328 7f31b130b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.328 7f31b130b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:31:07.328 7f31b130b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.328 7f31b130b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.328 7f31b130b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:07.328 7f31b130b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.389 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:31:07.396 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:31:07.397 7f31b12a8040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.398 7f31b12a8040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:07.398 7f31b12a8040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.398 7f31b12a8040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.398 7f31b12a8040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:07.398 7f31b12a8040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.399 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:07.400 7f31b130b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.401 7f31b130b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:07.401 7f31b130b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.401 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.401 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:07.402 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.402 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:07.403 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:31:07.403 7f31b130b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.403 7f31b130b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:07.403 7f31b130b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.404 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.410 7f31b12a8c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.411 7f31b12a8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:07.411 7f31b12a8c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.412 7f31b12a8c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.412 7f31b12a8c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:07.412 7f31b12a8c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.469 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:31:07.469 7f31b12a8ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.470 7f31b12a8ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:07.470 7f31b12a8ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.473 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:31:07.473 7f31b12a8ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.473 7f31b12a8ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:07.473 7f31b12a8ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.474 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:31:07.474 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:31:07.474 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:31:07.474 7f31b12a8e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:31:07.475 7f31b11e00e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.476 7f31b11e00e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:07.476 7f31b11e00e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.476 7f31b11e00e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.476 7f31b11e00e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:07.476 7f31b11e00e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.479 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:31:07.481 7f31b12a8e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:31:07.481 7f31b11e0860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.482 7f31b11e0860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:07.482 7f31b11e0860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.483 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:07.484 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.484 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:07.484 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.484 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:07.485 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.513 7f31b135b7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:31:07.515 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.515 7f31b11e0ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:31:07.515 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:07.660 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:07.660 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.661 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:07.661 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.661 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:07.662 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.946 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:07.947 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.947 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:07.948 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:07.948 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:07.948 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:08.327 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:08.379 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:08.380 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:08.380 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:08.380 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:08.381 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:08.381 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:09.329 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:09.391 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:09.392 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:09.392 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:09.393 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:09.393 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:09.393 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:10.330 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:11.321 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:31:11.323 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:31:11.323 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:31:11.323 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:31:11.324 7f31b11e0ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:31:11.324 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:31:11.324 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:31:11.324 7f31b11e0ea0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:31:11.324 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:31:11.324 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:31:11.325 7f31b11e0ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:31:11.325 7f31b3a3b180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:31:11.326 7f31b3a3b180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:31:11.330 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:11.892 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:11.893 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:11.893 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:11.894 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:11.894 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:11.894 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:12.331 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:13.332 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:14.334 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:15.335 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:16.336 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:16.623 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:16.624 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:16.624 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:16.625 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:16.625 7f31b130ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:16.626 7f31b130ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:17.338 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:18.339 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:19.340 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:20.365 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:21.366 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:22.352 7f31b130ba40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:31:22.352 7f31b130ba40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:31:22.353 7f31b130ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:31:22.367 7f31b135bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:22.548 7f31b135b7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:31:22.549 7f31b6128c00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:31:22.549 7f31b6128c00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:31:22.549 7f31b130b860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:31:22.549 7f31b130b860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:31:22.566 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:31:22.566 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:31:22.566 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:31:22.566 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:31:22.567 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:31:22.568 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:31:22.568 7f31b6128c00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:31:32.113 7f64964d2b00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:31:32.113 7f64964d2b00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:31:32.113 7f64964d2b00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:31:32.117 7f64964d2b00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:31:32.117 7f649180b2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:31:32.118 7f649180b2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:31:32.118 7f649180b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:31:32.118 7f649180b7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:31:32.118 7f649180b7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:31:32.119 7f649180b7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:31:32.119 7f649180b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.119 7f649180b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:31:32.119 7f649180b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.119 7f649180b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.119 7f649180b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353092118908 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.189:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:31:32.119 7f649180b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.120 7f649180b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.120 7f649180b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:32.120 7f649180b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.120 7f649180b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.121 7f649180b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:32.121 7f649180b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.121 7f649180b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.121 7f649180b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:32.121 7f649180b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.121 7f649180b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.121 7f649180b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:32.121 7f649180b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.121 7f649180be00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:31:32.122 7f649180be00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:31:32.122 7f649180bc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:31:32.123 7f64917bb540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.123 7f64917bb540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:31:32.123 7f64917bb540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.123 7f64917bb540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.123 7f64917bb540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:32.123 7f64917bb540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.183 7f649180b7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:31:32.191 7f649180b7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:31:32.192 7f6491755040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.193 7f6491755040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:32.193 7f6491755040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.193 7f6491755040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.193 7f6491755040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:32.193 7f6491755040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.194 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:32.195 7f64917bb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.196 7f64917bb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:32.196 7f64917bb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.197 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:32.197 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:32.198 7f649180b7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:31:32.198 7f64917bb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.198 7f64917bb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:32.198 7f64917bb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.199 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:32.199 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:32.206 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:32.207 7f6491755c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.207 7f6491755c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:32.207 7f6491755c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.209 7f6491755c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.209 7f6491755c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:32.209 7f6491755c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.245 7f649180b7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:31:32.245 7f6491755ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.246 7f6491755ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:32.246 7f6491755ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.251 7f649180b7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:31:32.251 7f6491755ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.252 7f6491755ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:32.252 7f6491755ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.252 7f649180b7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:31:32.252 7f649180b7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:31:32.252 7f649180b7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:31:32.253 7f6491755e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:31:32.253 7f649168e0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.254 7f649168e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:32.254 7f649168e0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.254 7f649168e0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.254 7f649168e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:32.254 7f649168e0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.258 7f649180b7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:31:32.260 7f6491755e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:31:32.260 7f649168e9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.260 7f649168e9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:32.260 7f649168e9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.296 7f649180b7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:31:32.298 7f649168e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.298 7f649168e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:31:32.298 7f649168e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:32.314 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:32.315 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:32.315 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:32.316 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:32.316 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:32.317 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:32.610 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:32.611 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:32.611 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:32.612 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:32.612 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:32.613 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:33.122 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:33.391 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:33.392 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:33.392 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:33.392 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:33.393 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:33.393 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:34.124 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:35.125 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:35.320 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:35.321 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:35.321 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:35.322 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:35.322 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:35.322 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:36.119 7f649168e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:31:36.120 7f649168e7c0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:31:36.120 7f649168e7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:31:36.120 7f649168e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:31:36.120 7f649168e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:31:36.120 7f649168e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:31:36.121 7f649168e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:31:36.121 7f649168e7c0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:31:36.121 7f649168e7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:31:36.121 7f649168e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:31:36.121 7f649168e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:31:36.121 7f6493ee6180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:31:36.122 7f6493ee6180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:31:36.126 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:37.127 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:38.129 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:39.130 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:39.350 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:39.351 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:39.351 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:39.351 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:39.352 7f64917bba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:39.352 7f64917bba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:40.131 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:41.134 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:42.135 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:43.136 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:44.137 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:45.138 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:46.139 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:47.140 7f649180bf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:47.177 7f64917bba40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:31:47.178 7f64917bba40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:31:47.178 7f64917bba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:31:47.362 7f649180b7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:31:47.362 7f64964d2b00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:31:47.362 7f64964d2b00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:31:47.362 7f64917bb860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:31:47.362 7f64917bb860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:31:47.385 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:31:47.386 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:31:47.386 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:31:47.386 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:31:47.386 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:31:47.386 7f64964d2b00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:31:57.488 7f216c05af80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:31:57.488 7f216c05af80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:31:57.488 7f216c05af80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:31:57.492 7f216c05af80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:31:57.493 7f21675d52c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:31:57.493 7f21675d52c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:31:57.493 7f21675d57c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:31:57.494 7f21675d57c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:31:57.494 7f21675d57c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:31:57.494 7f21675d57c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:31:57.494 7f21675d57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.495 7f21675d57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:31:57.495 7f21675d57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.495 7f21675d57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.495 7f21675d57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353117494486 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.50:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:31:57.495 7f21675d57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.496 7f21675d57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.497 7f21675d57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:57.497 7f21675d57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.497 7f21675d5e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:31:57.497 7f21675d5e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:31:57.498 7f21675d5c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:31:57.499 7f2167585540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.499 7f2167585540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:31:57.499 7f2167585540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.499 7f2167585540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.499 7f2167585540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:57.499 7f2167585540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.554 7f21675d57c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:31:57.562 7f21675d57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:31:57.563 7f21696e60e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.563 7f21696e60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:57.563 7f21696e60e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.563 7f21696e60e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.563 7f21696e60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:57.563 7f21696e60e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.564 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:57.565 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:57.565 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:57.566 7f21675857c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.566 7f21675857c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:57.566 7f21675857c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.567 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:57.567 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:57.568 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:57.568 7f21675d57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:31:57.568 7f21675857c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.569 7f21675857c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:57.569 7f21675857c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.570 7f21696e60e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.570 7f21696e60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:57.570 7f21696e60e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.577 7f21696e6d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.577 7f21696e6d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:57.577 7f21696e6d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.602 7f21675d57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:31:57.603 7f21696e6c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.603 7f21696e6c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:31:57.603 7f21696e6c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.611 7f21675d57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:31:57.612 7f21696e6c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.612 7f21696e6c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:31:57.612 7f21696e6c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.612 7f21675d57c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:31:57.613 7f21675d57c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:31:57.613 7f21675d57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:31:57.613 7f21696e6f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:31:57.614 7f2168bad220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.614 7f2168bad220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:57.614 7f2168bad220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.614 7f2168bad220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.614 7f2168bad220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:57.614 7f2168bad220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.617 7f21675d57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:31:57.619 7f21696e6f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:31:57.619 7f2168bad9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.619 7f2168bad9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:31:57.620 7f2168bad9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.651 7f21675d57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:31:57.653 7f2168b89400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.653 7f2168b89400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:31:57.653 7f2168b89400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:31:57.664 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:57.665 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:57.665 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:57.665 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:57.665 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:57.666 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:57.931 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:57.932 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:57.932 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:57.933 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:57.933 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:57.933 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:58.498 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:58.553 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:58.554 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:58.554 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:58.555 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:58.555 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:58.555 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:59.499 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:31:59.690 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:31:59.691 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:59.691 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:31:59.691 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:31:59.691 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:31:59.692 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:00.500 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:01.494 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:32:01.496 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:32:01.496 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:32:01.496 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:32:01.496 7f2168bad5e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:32:01.496 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:32:01.496 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:32:01.496 7f2168bad5e0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:32:01.497 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:32:01.497 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:32:01.497 7f2168bad5e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:32:01.497 7f2168badc20 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:32:01.498 7f2168badc20 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:32:01.501 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:02.335 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:02.336 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:02.337 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:02.337 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:02.337 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:02.338 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:02.502 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:03.503 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:04.504 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:05.506 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:06.507 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:07.509 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:07.861 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:07.861 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:07.861 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:07.862 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:07.862 7f2167585c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:07.863 7f2167585c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:08.510 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:09.512 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:10.513 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:11.514 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:12.514 7f21675d5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:12.585 7f2167585c20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:32:12.585 7f2167585c20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:32:12.586 7f2167585c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:32:12.674 7f21675d57c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:32:12.674 7f216c05af80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:32:12.675 7f216c05af80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:32:12.675 7f2167585b80 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:32:12.675 7f2167585b80 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:32:12.689 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:32:12.690 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:32:12.691 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:32:12.691 7f216c05af80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:32:21.744 7f0e5303cf80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:32:21.744 7f0e5303cf80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:32:21.744 7f0e5303cf80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:32:21.747 7f0e5303cf80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:32:21.748 7f0e4e2bf2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:32:21.748 7f0e4e2bf2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:32:21.748 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:32:21.748 7f0e4e2bf7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:32:21.749 7f0e4e2bf7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:32:21.749 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:32:21.749 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.749 7f0e4e2bf7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:32:21.749 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.749 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.749 7f0e4e2bf7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353141749027 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:32:21.749 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.750 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.750 7f0e4e2bf7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:32:21.750 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:21.751 7f0e4e2bf7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.751 7f0e4e2bfe00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:32:21.752 7f0e4e2bfe00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:32:21.753 7f0e4e2bfc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:32:21.753 7f0e4e26f540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.754 7f0e4e26f540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:32:21.754 7f0e4e26f540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.754 7f0e4e26f540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.754 7f0e4e26f540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:21.754 7f0e4e26f540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.816 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:32:21.824 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:32:21.825 7f0e4e20c0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.825 7f0e4e20c0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:32:21.825 7f0e4e20c0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.825 7f0e4e20c0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.826 7f0e4e20c0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:32:21.826 7f0e4e20c0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.827 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:21.828 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:21.828 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:21.828 7f0e4e26f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.829 7f0e4e26f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:32:21.829 7f0e4e26f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.830 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:21.830 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:21.831 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:32:21.831 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:21.831 7f0e4e26f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.831 7f0e4e26f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:21.831 7f0e4e26f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.832 7f0e4e20c0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.833 7f0e4e20c0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:21.833 7f0e4e20c0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.840 7f0e4e20cd60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.840 7f0e4e20cd60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:21.840 7f0e4e20cd60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.866 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:32:21.867 7f0e4e20cc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.868 7f0e4e20cc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:32:21.868 7f0e4e20cc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.871 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:32:21.871 7f0e4e20cc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.872 7f0e4e20cc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:21.872 7f0e4e20cc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.872 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:32:21.872 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:32:21.872 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:32:21.873 7f0e4e20cf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:32:21.873 7f0e4e140220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.873 7f0e4e140220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:32:21.874 7f0e4e140220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.874 7f0e4e140220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.874 7f0e4e140220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:32:21.874 7f0e4e140220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.877 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:32:21.879 7f0e4e20cf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:32:21.879 7f0e4e1409a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.879 7f0e4e1409a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:32:21.879 7f0e4e1409a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.910 7f0e4e2bf7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:32:21.912 7f0e4f953400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.912 7f0e4f953400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:32:21.912 7f0e4f953400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:21.923 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:21.924 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:21.924 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:21.924 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:21.924 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:21.925 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.077 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:22.078 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.078 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:22.079 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.079 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:22.079 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.418 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:22.419 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.419 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:22.420 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.420 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:22.420 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.753 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:22.839 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:22.839 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.839 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:22.840 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:22.840 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:22.841 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:23.756 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:23.865 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:23.865 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:23.866 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:23.866 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:23.866 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:23.867 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:24.758 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:25.749 7f0e4f953400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:32:25.751 7f0e4f953400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:32:25.751 7f0e4f953400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:32:25.751 7f0e4f953400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:32:25.751 7f0e4f953400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:32:25.751 7f0e4f953400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:32:25.751 7f0e4f953400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:32:25.751 7f0e4f953400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:32:25.752 7f0e4f953400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:32:25.752 7f0e4f953400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:32:25.752 7f0e4f953400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:32:25.752 7f0e4f953900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:32:25.753 7f0e4f953900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:32:25.758 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:26.430 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:26.431 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:26.431 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:26.432 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:26.432 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:26.432 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:26.759 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:27.761 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:28.762 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:29.762 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:30.763 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:31.542 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:31.543 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:31.543 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:31.544 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:31.544 7f0e4e26fc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:31.545 7f0e4e26fc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:31.765 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:32.765 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:33.767 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:34.768 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:35.769 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:36.767 7f0e4e26fc20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:32:36.767 7f0e4e26fc20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:32:36.768 7f0e4e26fc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:32:36.770 7f0e4e2bff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:36.968 7f0e4e2bf7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:32:36.968 7f0e5303cf80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:32:36.969 7f0e5303cf80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:32:36.969 7f0e4e26f680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:32:36.969 7f0e4e26f680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:32:36.983 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:32:36.983 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:32:36.983 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:32:36.983 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:32:36.983 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:32:36.984 7f0e5303cf80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:32:46.517 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:32:46.517 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:32:46.517 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:32:46.521 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:32:46.521 7f0f6e0862c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:32:46.521 7f0f6e0862c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:32:46.521 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:32:46.522 7f0f6e0867c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:32:46.522 7f0f6e0867c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:32:46.522 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:32:46.523 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.523 7f0f6e0867c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:32:46.523 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.523 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.523 7f0f6e0867c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353166522567 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:32:46.523 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.524 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.524 7f0f6e0867c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:32:46.524 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.524 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.524 7f0f6e0867c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:32:46.524 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.524 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.525 7f0f6e0867c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:32:46.525 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.525 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.525 7f0f6e0867c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:46.525 7f0f6e0867c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.525 7f0f6e086e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:32:46.525 7f0f6e086e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:32:46.526 7f0f6e086c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:32:46.527 7f0f6e036540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.527 7f0f6e036540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:32:46.527 7f0f6e036540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.527 7f0f6e036540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.527 7f0f6e036540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:46.527 7f0f6e036540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.594 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:32:46.603 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:32:46.604 7f0f6dfe4040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.604 7f0f6dfe4040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:32:46.604 7f0f6dfe4040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.604 7f0f6dfe4040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.604 7f0f6dfe4040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:32:46.605 7f0f6dfe4040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.606 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:46.607 7f0f6e0367c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.607 7f0f6e0367c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:32:46.607 7f0f6e0367c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.608 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:46.608 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:46.609 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:46.609 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:46.610 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:46.610 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:32:46.610 7f0f6e0367c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.610 7f0f6e0367c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:46.610 7f0f6e0367c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.619 7f0f6dfe4040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.619 7f0f6dfe4040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:46.619 7f0f6dfe4040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.621 7f0f6dfe4040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.621 7f0f6dfe4040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:46.621 7f0f6dfe4040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.651 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:32:46.652 7f0f6dfe4c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.653 7f0f6dfe4c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:32:46.653 7f0f6dfe4c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.657 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:32:46.657 7f0f6dfe4c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.657 7f0f6dfe4c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:32:46.657 7f0f6dfe4c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.658 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:32:46.658 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:32:46.658 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:32:46.658 7f0f6dfe4e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:32:46.659 7f0f701740e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.660 7f0f701740e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:32:46.660 7f0f701740e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.660 7f0f701740e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.660 7f0f701740e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:32:46.660 7f0f701740e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.664 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:32:46.668 7f0f6dfe4e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:32:46.668 7f0f70174c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.669 7f0f70174c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:32:46.669 7f0f70174c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.699 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:46.700 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:46.700 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:46.701 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:46.701 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:46.701 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:46.704 7f0f6e0867c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:32:46.705 7f0f70174900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.705 7f0f70174900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:32:46.705 7f0f70174900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:32:46.938 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:46.938 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:46.939 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:46.939 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:46.939 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:46.940 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:47.253 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:47.254 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:47.254 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:47.254 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:47.254 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:47.255 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:47.526 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:47.692 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:47.693 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:47.694 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:47.694 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:47.694 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:47.695 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:48.258 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:48.259 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:48.259 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:48.259 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:48.260 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:48.260 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:48.527 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:49.131 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:49.132 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:49.132 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:49.133 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:49.133 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:49.133 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:49.528 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:50.523 7f0f70174900 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:32:50.524 7f0f70174900 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:32:50.524 7f0f70174900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:32:50.525 7f0f70174900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:32:50.525 7f0f70174900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:32:50.525 7f0f70174900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:32:50.525 7f0f70174900 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:32:50.525 7f0f70174900 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:32:50.525 7f0f70174900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:32:50.526 7f0f70174900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:32:50.526 7f0f70174900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:32:50.526 7f0f6f63a400 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:32:50.527 7f0f6f63a400 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:32:50.529 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:50.552 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:50.553 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:50.553 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:50.553 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:50.553 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:50.554 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:51.530 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:52.333 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:52.334 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:52.334 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:52.335 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:52.335 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:52.336 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:52.531 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:53.531 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:54.532 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:55.047 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:32:55.048 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:55.049 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:32:55.049 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:55.049 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:32:55.049 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:32:55.533 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:56.534 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:57.535 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:58.536 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:32:59.536 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:00.405 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:00.406 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:00.406 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:00.407 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:00.407 7f0f6e036a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:00.407 7f0f6e036a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:00.537 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:01.538 7f0f6e086f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:01.612 7f0f6e036a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:33:01.612 7f0f6e036a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:33:01.613 7f0f6e036a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:33:01.741 7f0f6e0867c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:33:01.741 7f0f72b0ad00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:33:01.741 7f0f72b0ad00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:33:01.741 7f0f6e036860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:33:01.742 7f0f6e036860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:33:01.756 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:33:01.757 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:33:01.758 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:33:01.758 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:33:01.758 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:33:01.758 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:33:01.758 7f0f72b0ad00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:33:11.108 7f14e284fc40 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:33:11.108 7f14e284fc40 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:33:11.108 7f14e284fc40 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:33:11.111 7f14e284fc40 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:33:11.112 7f14dd7ef2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:33:11.112 7f14dd7ef2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:33:11.112 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:33:11.113 7f14dd7ef7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:33:11.113 7f14dd7ef7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:33:11.113 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:33:11.113 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.113 7f14dd7ef7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:33:11.114 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.114 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.114 7f14dd7ef7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353191113539 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:33:11.114 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.115 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.115 7f14dd7ef7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:33:11.115 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.115 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.115 7f14dd7ef7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:33:11.115 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.115 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.115 7f14dd7ef7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:33:11.116 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.116 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.116 7f14dd7ef7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:11.116 7f14dd7ef7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.116 7f14dd7efe00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:33:11.116 7f14dd7efe00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:33:11.117 7f14dd7efc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:33:11.118 7f14dd79f540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.118 7f14dd79f540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:33:11.118 7f14dd79f540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.118 7f14dd79f540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.118 7f14dd79f540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:11.118 7f14dd79f540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.177 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:33:11.185 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:33:11.186 7f14df907040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.186 7f14df907040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:11.186 7f14df907040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.186 7f14df907040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.186 7f14df907040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:11.187 7f14df907040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.188 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:11.189 7f14dd79f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.189 7f14dd79f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:11.189 7f14dd79f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.189 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.190 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:11.190 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.190 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:11.191 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.192 7f14dd79f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.192 7f14dd79f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:11.192 7f14dd79f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.193 7f14dd79f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.193 7f14dd79f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:11.193 7f14dd79f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.200 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:33:11.200 7f14dd79f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.200 7f14dd79f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:11.200 7f14dd79f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.235 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:33:11.236 7f14df907ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.236 7f14df907ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:11.236 7f14df907ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.239 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:33:11.240 7f14df907ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.240 7f14df907ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:11.240 7f14df907ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.240 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:33:11.240 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:33:11.241 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:33:11.241 7f14df907e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:33:11.241 7f14dd7770e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.242 7f14dd7770e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:33:11.242 7f14dd7770e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.242 7f14dd7770e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.242 7f14dd7770e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:33:11.242 7f14dd7770e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.245 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:33:11.248 7f14df907e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:33:11.248 7f14dd777860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.248 7f14dd777860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:33:11.248 7f14dd777860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.272 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:11.273 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.273 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:11.274 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.274 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:11.274 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.276 7f14dd7ef7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:33:11.277 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.277 7f14dd777ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:33:11.277 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:11.447 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:11.448 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.448 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:11.448 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.448 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:11.449 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.914 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:11.915 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.915 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:11.916 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:11.916 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:11.916 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:12.117 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:12.961 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:12.962 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:12.962 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:12.963 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:12.963 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:12.964 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:13.118 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:14.120 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:14.738 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:14.739 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:14.739 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:14.740 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:14.740 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:14.740 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:15.113 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:33:15.115 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:33:15.115 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:33:15.115 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:33:15.115 7f14dd777ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:33:15.115 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:33:15.115 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:33:15.115 7f14dd777ea0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:33:15.116 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:33:15.116 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:33:15.116 7f14dd777ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:33:15.116 7f14dd1032c0 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:33:15.117 7f14dd1032c0 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:33:15.121 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:16.122 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:17.123 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:18.124 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:18.270 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:18.270 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:18.271 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:18.271 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:18.271 7f14dd79fa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:18.272 7f14dd79fa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:19.126 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:20.127 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:21.128 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:22.128 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:23.130 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:24.131 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:25.132 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:26.133 7f14dd7eff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:26.219 7f14dd79fa40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:33:26.219 7f14dd79fa40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:33:26.219 7f14dd79fa40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:33:26.297 7f14dd7ef7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:33:26.297 7f14e284fc40 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:33:26.297 7f14e284fc40 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:33:26.298 7f14dd79f860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:33:26.298 7f14dd79f860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:33:26.313 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:33:26.313 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:33:26.313 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:33:26.313 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:33:26.314 7f14e284fc40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:33:35.899 7efcbd437c80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:33:35.900 7efcbd437c80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:33:35.900 7efcbd437c80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:33:35.903 7efcbd437c80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:33:35.904 7efcb87d92c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:33:35.904 7efcb87d92c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:33:35.904 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:33:35.905 7efcb87d97c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:33:35.905 7efcb87d97c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:33:35.905 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:33:35.905 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.906 7efcb87d97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:33:35.906 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.906 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.906 7efcb87d97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353215905671 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:33:35.906 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.907 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.907 7efcb87d97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:33:35.907 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.907 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.907 7efcb87d97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:33:35.907 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.907 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.907 7efcb87d97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:33:35.908 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.908 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.908 7efcb87d97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:35.908 7efcb87d97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.908 7efcb87d9f40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:33:35.908 7efcb87d9f40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:33:35.909 7efcb87d9e00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:33:35.910 7efcb87897c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.910 7efcb87897c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:33:35.910 7efcb87897c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.910 7efcb87897c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.910 7efcb87897c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:35.910 7efcb87897c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.969 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:33:35.978 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:33:35.979 7efcb86d60e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.980 7efcb86d60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:35.980 7efcb86d60e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.980 7efcb86d60e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.980 7efcb86d60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:35.980 7efcb86d60e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.982 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:35.983 7efcb87899a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.983 7efcb87899a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:35.983 7efcb87899a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.984 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:35.984 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:35.987 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:35.987 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:35.988 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:35.991 7efcb87899a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.991 7efcb87899a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:35.991 7efcb87899a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.994 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:33:35.994 7efcb87899a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.994 7efcb87899a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:35.994 7efcb87899a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.995 7efcb86d60e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:35.996 7efcb86d60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:35.996 7efcb86d60e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.029 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:33:36.030 7efcb86d6c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.030 7efcb86d6c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:33:36.030 7efcb86d6c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.034 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:33:36.034 7efcb86d6c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.034 7efcb86d6c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:33:36.034 7efcb86d6c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.037 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:33:36.037 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:33:36.037 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:33:36.037 7efcb86d6f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:33:36.038 7efcb8747220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.038 7efcb8747220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:33:36.038 7efcb8747220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.039 7efcb8747220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.039 7efcb8747220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:33:36.039 7efcb8747220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.043 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:33:36.045 7efcb86d6f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:33:36.045 7efcb87479a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.045 7efcb87479a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:33:36.045 7efcb87479a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.057 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:36.058 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.058 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:36.059 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.059 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:36.060 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.092 7efcb87d97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:33:36.094 7efcb87477c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.095 7efcb87477c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:33:36.095 7efcb87477c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:33:36.166 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:36.167 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.167 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:36.168 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.168 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:36.168 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.323 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:36.324 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.325 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:36.325 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.325 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:36.326 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.644 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:36.645 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.646 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:36.646 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.646 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:36.647 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:36.909 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:37.104 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:37.105 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:37.106 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:37.106 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:37.106 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:37.107 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:37.769 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:37.770 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:37.770 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:37.771 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:37.771 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:37.771 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:37.910 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:38.911 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:39.515 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:39.516 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:39.516 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:39.517 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:39.517 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:39.517 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:39.905 7efcb87477c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:33:39.906 7efcb87477c0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:33:39.906 7efcb87477c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:33:39.906 7efcb87477c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:33:39.907 7efcb87477c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:33:39.907 7efcb87477c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:33:39.907 7efcb87477c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:33:39.907 7efcb87477c0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:33:39.907 7efcb87477c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:33:39.907 7efcb87477c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:33:39.908 7efcb87477c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:33:39.908 7efcbaec7180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:33:39.909 7efcbaec7180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:33:39.913 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:40.914 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:41.915 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:42.917 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:43.624 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:33:43.626 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:43.626 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:33:43.627 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:43.627 7efcb8789c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:33:43.628 7efcb8789c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:33:43.918 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:44.920 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:45.921 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:46.923 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:47.924 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:48.925 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:49.927 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:50.927 7efcb87892c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:33:50.968 7efcb8789c20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:33:50.969 7efcb8789c20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:33:50.969 7efcb8789c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:33:51.103 7efcb87d97c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:33:51.103 7efcbd437c80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:33:51.103 7efcbd437c80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:33:51.104 7efcb8789b80 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:33:51.104 7efcb8789b80 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:33:51.117 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:33:51.118 7efcbd437c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:34:00.257 7fdc582ed700 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:34:00.258 7fdc582ed700 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:34:00.258 7fdc582ed700 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:34:00.261 7fdc582ed700 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:34:00.262 7fdc538692c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:34:00.262 7fdc538692c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:34:00.262 7fdc538697c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:34:00.263 7fdc538697c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:34:00.263 7fdc538697c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:34:00.263 7fdc538697c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:34:00.263 7fdc538697c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.264 7fdc538697c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:34:00.264 7fdc538697c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.264 7fdc538697c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.264 7fdc538697c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353240263623 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:34:00.264 7fdc538697c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.265 7fdc538697c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.265 7fdc538697c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:00.265 7fdc538697c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.265 7fdc538697c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.265 7fdc538697c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:00.265 7fdc538697c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.265 7fdc538697c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.266 7fdc538697c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:00.266 7fdc538697c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.266 7fdc538697c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.266 7fdc538697c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:00.266 7fdc538697c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.266 7fdc53869e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:34:00.267 7fdc53869e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:34:00.268 7fdc53869c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:34:00.268 7fdc53819540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.268 7fdc53819540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:34:00.268 7fdc53819540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.268 7fdc53819540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.269 7fdc53819540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:00.269 7fdc53819540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.329 7fdc538697c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:34:00.337 7fdc538697c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:34:00.338 7fdc537a3040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.338 7fdc537a3040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:00.339 7fdc537a3040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.339 7fdc537a3040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.339 7fdc537a3040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:00.339 7fdc537a3040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.340 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:00.341 7fdc538197c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.341 7fdc538197c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:00.341 7fdc538197c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.342 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:00.342 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:00.343 7fdc538697c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:34:00.344 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:00.344 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:00.344 7fdc538197c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.344 7fdc538197c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:00.344 7fdc538197c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.345 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:00.352 7fdc537a3c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.353 7fdc537a3c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:00.353 7fdc537a3c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.354 7fdc537a3c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.354 7fdc537a3c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:00.354 7fdc537a3c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.384 7fdc538697c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:34:00.385 7fdc537a3ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.386 7fdc537a3ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:00.386 7fdc537a3ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.389 7fdc538697c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:34:00.390 7fdc537a3ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.390 7fdc537a3ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:00.390 7fdc537a3ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.390 7fdc538697c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:34:00.390 7fdc538697c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:34:00.391 7fdc538697c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:34:00.391 7fdc537a3e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:34:00.392 7fdc537740e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.392 7fdc537740e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:00.392 7fdc537740e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.392 7fdc537740e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.393 7fdc537740e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:00.393 7fdc537740e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.396 7fdc538697c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:34:00.398 7fdc537a3e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:34:00.398 7fdc537749a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.399 7fdc537749a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:00.399 7fdc537749a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.422 7fdc538697c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:34:00.424 7fdc53703400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.424 7fdc53703400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:34:00.425 7fdc53703400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:00.462 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:00.462 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:00.463 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:00.463 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:00.463 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:00.463 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:00.703 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:00.704 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:00.704 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:00.704 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:00.704 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:00.705 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:01.065 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:01.066 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:01.066 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:01.067 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:01.067 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:01.067 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:01.267 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:01.726 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:01.727 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:01.727 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:01.728 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:01.728 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:01.728 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:02.268 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:03.066 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:03.066 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:03.066 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:03.067 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:03.067 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:03.068 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:03.270 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:04.263 7fdc53703400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:34:04.264 7fdc53703400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:34:04.265 7fdc53703400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:34:04.265 7fdc53703400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:34:04.265 7fdc53703400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:34:04.265 7fdc53703400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:34:04.265 7fdc53703400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:34:04.265 7fdc53703400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:34:04.266 7fdc53703400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:34:04.266 7fdc53703400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:34:04.266 7fdc53703400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:34:04.266 7fdc53703900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:34:04.268 7fdc53703900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:34:04.271 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:05.272 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:06.273 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:06.732 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:06.733 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:06.733 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:06.733 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:06.733 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:06.734 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:07.274 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:08.275 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:09.276 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:10.278 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:11.279 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:12.281 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:12.508 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:12.509 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:12.509 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:12.509 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:12.509 7fdc53819a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:12.510 7fdc53819a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:13.282 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:14.283 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:15.285 7fdc53869f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:15.322 7fdc53819a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:34:15.322 7fdc53819a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:34:15.323 7fdc53819a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:34:15.492 7fdc538697c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:34:15.493 7fdc582ed700 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:34:15.493 7fdc582ed700 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:34:15.493 7fdc53819860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:34:15.493 7fdc53819860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:34:15.507 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:34:15.508 7fdc582ed700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:34:25.094 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:34:25.094 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:34:25.095 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:34:25.099 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:34:25.099 7f31ac06d2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:34:25.099 7f31ac06d2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:34:25.100 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:34:25.100 7f31ac06d7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:34:25.100 7f31ac06d7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:34:25.101 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:34:25.101 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.101 7f31ac06d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:34:25.101 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.101 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.101 7f31ac06d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353265100731 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:34:25.102 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.102 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:25.103 7f31ac06d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.104 7f31ac06de00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:34:25.104 7f31ac06de00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:34:25.105 7f31ac06dc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:34:25.106 7f31ac01d540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.106 7f31ac01d540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:34:25.106 7f31ac01d540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.106 7f31ac01d540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.106 7f31ac01d540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:25.106 7f31ac01d540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.171 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:34:25.179 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:34:25.180 7f31abf80040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.180 7f31abf80040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:25.181 7f31abf80040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.181 7f31abf80040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.181 7f31abf80040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:25.181 7f31abf80040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.182 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:25.183 7f31ac01d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.183 7f31ac01d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:25.184 7f31ac01d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.184 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:25.184 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:25.185 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:25.185 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:25.185 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:34:25.186 7f31ac01d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.186 7f31ac01d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:25.186 7f31ac01d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.186 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:25.194 7f31abf80c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.194 7f31abf80c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:25.194 7f31abf80c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.196 7f31abf80c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.196 7f31abf80c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:25.196 7f31abf80c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.223 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:34:25.223 7f31abf80ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.224 7f31abf80ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:25.224 7f31abf80ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.227 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:34:25.227 7f31abf80ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.228 7f31abf80ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:25.228 7f31abf80ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.228 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:34:25.228 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:34:25.229 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:34:25.229 7f31abf80e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:34:25.230 7f31abee90e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.230 7f31abee90e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:25.230 7f31abee90e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.230 7f31abee90e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.230 7f31abee90e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:25.230 7f31abee90e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.234 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:34:25.235 7f31abf80e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:34:25.236 7f31abee99a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.236 7f31abee99a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:25.236 7f31abee99a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.260 7f31ac06d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:34:25.263 7f31ad702400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.263 7f31ad702400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:34:25.263 7f31ad702400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:25.296 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:25.297 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:25.297 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:25.298 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:25.298 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:25.299 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:25.577 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:25.578 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:25.578 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:25.579 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:25.579 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:25.579 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:26.105 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:26.302 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:26.303 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:26.303 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:26.304 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:26.304 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:26.304 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:27.106 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:27.539 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:27.540 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:27.540 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:27.541 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:27.541 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:27.541 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:28.107 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:29.100 7f31ad702400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:34:29.102 7f31ad702400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:34:29.102 7f31ad702400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:34:29.102 7f31ad702400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:34:29.103 7f31ad702400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:34:29.103 7f31ad702400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:34:29.103 7f31ad702400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:34:29.103 7f31ad702400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:34:29.104 7f31ad702400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:34:29.104 7f31ad702400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:34:29.104 7f31ad702400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:34:29.104 7f31ad702900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:34:29.105 7f31ad702900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:34:29.108 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:29.530 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:29.530 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:29.531 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:29.531 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:29.531 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:29.532 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:30.108 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:31.110 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:32.111 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:32.666 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:32.667 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:32.667 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:32.668 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:32.668 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:32.668 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:33.112 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:34.114 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:35.115 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:36.116 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:36.990 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:36.990 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:36.991 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:36.991 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:36.991 7f31ac01da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:36.991 7f31ac01da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:37.117 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:38.119 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:39.119 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:40.121 7f31ac06df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:40.207 7f31ac01da40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:34:40.207 7f31ac01da40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:34:40.208 7f31ac01da40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:34:40.284 7f31ac06d7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:34:40.284 7f31b0e3ce80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:34:40.285 7f31b0e3ce80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:34:40.285 7f31ac01d680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:34:40.285 7f31ac01d680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:34:40.299 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:34:40.300 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:34:40.301 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:34:40.301 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:34:40.301 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:34:40.301 7f31b0e3ce80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:34:49.060 7f6564431040 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:34:49.060 7f6564431040 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:34:49.060 7f6564431040 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:34:49.063 7f6564431040 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:34:49.063 7f655f2932c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:34:49.063 7f655f2932c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:34:49.064 7f655f2937c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:34:49.064 7f655f2937c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:34:49.064 7f655f2937c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:34:49.064 7f655f2937c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:34:49.065 7f655f2937c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.065 7f655f2937c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:34:49.065 7f655f2937c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.065 7f655f2937c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.065 7f655f2937c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353289064751 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:34:49.065 7f655f2937c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:49.066 7f655f2937c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.067 7f655f293e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:34:49.067 7f655f293e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:34:49.068 7f655f293c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:34:49.068 7f655f242540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.068 7f655f242540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:34:49.068 7f655f242540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.069 7f655f242540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.069 7f655f242540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:49.069 7f655f242540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.122 7f655f2937c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:34:49.129 7f655f2937c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:34:49.130 7f65613a3040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.131 7f65613a3040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:49.131 7f65613a3040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.131 7f65613a3040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.131 7f65613a3040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:49.131 7f65613a3040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.132 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:49.132 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.133 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:49.133 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.133 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:49.133 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.135 7f655f2427c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.135 7f655f2427c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:49.135 7f655f2427c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.137 7f655f2937c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:34:49.137 7f655f2427c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.137 7f655f2427c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:49.137 7f655f2427c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.146 7f65613a3c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.146 7f65613a3c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:49.146 7f65613a3c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.148 7f65613a3c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.148 7f65613a3c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:49.148 7f65613a3c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.177 7f655f2937c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:34:49.178 7f65613a3ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.179 7f65613a3ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:34:49.179 7f65613a3ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.182 7f655f2937c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:34:49.183 7f65613a3ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.183 7f65613a3ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:34:49.183 7f65613a3ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.183 7f655f2937c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:34:49.183 7f655f2937c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:34:49.184 7f655f2937c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:34:49.184 7f65613a3e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:34:49.185 7f656086d0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.185 7f656086d0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:49.185 7f656086d0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.185 7f656086d0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.185 7f656086d0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:49.185 7f656086d0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.189 7f655f2937c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:34:49.191 7f65613a3e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:34:49.191 7f656086d9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.191 7f656086d9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:34:49.191 7f656086d9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.198 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:49.198 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.198 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:49.202 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.202 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:49.203 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.216 7f655f2937c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:34:49.218 7f6560843400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.218 7f6560843400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:34:49.218 7f6560843400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:34:49.332 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:49.333 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.333 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:49.333 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.333 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:49.334 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.541 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:49.542 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.542 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:49.543 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:49.543 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:49.543 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:50.044 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:50.045 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:50.045 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:50.045 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:50.045 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:50.046 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:50.067 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:51.068 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:51.159 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:51.160 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:51.160 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:51.160 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:51.160 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:51.161 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:52.070 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:52.646 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:52.646 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:52.646 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:52.648 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:52.648 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:52.648 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:53.065 7f6560843400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:34:53.066 7f6560843400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:34:53.066 7f6560843400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:34:53.066 7f6560843400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:34:53.066 7f6560843400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:34:53.066 7f6560843400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:34:53.066 7f6560843400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:34:53.067 7f6560843400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:34:53.067 7f6560843400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:34:53.067 7f6560843400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:34:53.067 7f6560843400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:34:53.067 7f6560843900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:34:53.068 7f6560843900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:34:53.071 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:54.071 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:55.072 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:56.073 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:56.376 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:34:56.377 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:56.377 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:34:56.378 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:56.378 7f655f242a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:34:56.379 7f655f242a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:34:57.074 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:58.075 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:34:59.077 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:00.078 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:01.079 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:02.080 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:03.081 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:04.082 7f655f293f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:04.114 7f655f242a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:35:04.114 7f655f242a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:35:04.115 7f655f242a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:35:04.284 7f655f2937c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:35:04.284 7f6564431040 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:35:04.285 7f6564431040 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:35:04.285 7f655f242860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:35:04.285 7f655f242860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:35:04.299 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:35:04.299 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:35:04.299 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:35:04.299 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:35:04.299 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:35:04.299 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:35:04.299 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:35:04.299 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:35:04.300 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:35:04.300 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:35:04.300 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:35:04.300 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:35:04.300 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:35:04.300 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:35:04.300 7f6564431040 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:35:13.444 7f514803eac0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:35:13.444 7f514803eac0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:35:13.444 7f514803eac0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:35:13.454 7f514803eac0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:35:13.457 7f5142fdb2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:35:13.458 7f5142fdb2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:35:13.458 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:35:13.458 7f5142fdb7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:35:13.459 7f5142fdb7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:35:13.459 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:35:13.459 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.459 7f5142fdb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:35:13.459 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.459 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.459 7f5142fdb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353313459010 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:35:13.459 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.460 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.460 7f5142fdb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:35:13.460 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.460 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.460 7f5142fdb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:35:13.460 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.460 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.461 7f5142fdb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:35:13.461 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.461 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.461 7f5142fdb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:13.461 7f5142fdb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.461 7f5142fdbe00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:35:13.461 7f5142fdbe00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:35:13.463 7f5142fdbc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:35:13.465 7f5142f8b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.466 7f5142f8b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:35:13.466 7f5142f8b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.466 7f5142f8b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.466 7f5142f8b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:13.466 7f5142f8b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.616 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:35:13.624 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:35:13.625 7f5142f71040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.626 7f5142f71040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:35:13.626 7f5142f71040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.626 7f5142f71040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.626 7f5142f71040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:35:13.626 7f5142f71040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.627 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:13.627 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:13.627 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:13.628 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:13.628 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:13.629 7f5142f8b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.630 7f5142f8b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:35:13.630 7f5142f8b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.630 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:13.631 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:35:13.631 7f5142f8b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.631 7f5142f8b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:13.631 7f5142f8b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.639 7f5142f71c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.640 7f5142f71c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:13.640 7f5142f71c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.641 7f5142f71c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.641 7f5142f71c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:13.642 7f5142f71c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.675 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:35:13.675 7f5142f71ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.676 7f5142f71ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:35:13.676 7f5142f71ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.679 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:35:13.679 7f5142f71ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.679 7f5142f71ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:13.679 7f5142f71ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.680 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:35:13.680 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:35:13.680 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:35:13.680 7f5142f71e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:35:13.681 7f51445ae0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.681 7f51445ae0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:35:13.681 7f51445ae0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.681 7f51445ae0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.681 7f51445ae0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:35:13.681 7f51445ae0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.684 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:35:13.697 7f5142f71e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:35:13.698 7f51445ae860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.698 7f51445ae860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:35:13.698 7f51445ae860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.720 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:13.721 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:13.721 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:13.722 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:13.722 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:13.723 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:13.723 7f5142fdb7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:35:13.725 7f51445aef40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.725 7f51445aef40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:35:13.725 7f51445aef40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:13.913 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:13.914 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:13.914 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:13.915 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:13.915 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:13.915 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:14.321 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:14.321 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:14.321 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:14.322 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:14.322 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:14.323 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:14.462 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:15.211 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:15.228 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:15.228 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:15.229 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:15.229 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:15.229 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:15.463 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:16.464 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:16.736 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:16.737 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:16.737 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:16.737 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:16.737 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:16.738 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:17.459 7f51445aef40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:35:17.460 7f51445aef40 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:35:17.461 7f51445aef40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:35:17.461 7f51445aef40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:35:17.461 7f51445aef40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:35:17.461 7f51445aef40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:35:17.461 7f51445aef40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:35:17.461 7f51445aef40 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:35:17.461 7f51445aef40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:35:17.462 7f51445aef40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:35:17.462 7f51445aef40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:35:17.462 7f514458a360 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:35:17.463 7f514458a360 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:35:17.466 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:18.467 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:19.468 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:20.108 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:20.109 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:20.109 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:20.110 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:20.110 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:20.110 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:20.469 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:21.470 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:22.471 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:23.472 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:24.474 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:25.475 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:25.856 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:25.857 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:25.857 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:25.858 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:25.858 7f5142f8ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:25.858 7f5142f8ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:26.476 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:27.477 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:28.470 7f5142f8ba40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:35:28.470 7f5142f8ba40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:35:28.471 7f5142f8ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:35:28.479 7f5142fdbf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:28.823 7f5142fdb7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:35:28.823 7f514803eac0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:35:28.823 7f514803eac0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:35:28.824 7f5142f8b860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:35:28.824 7f5142f8b860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:35:28.842 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:35:28.842 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:35:28.842 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:35:28.842 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:35:28.842 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:35:28.842 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:35:28.842 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:35:28.843 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:35:28.843 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:35:28.843 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:35:28.843 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:35:28.843 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:35:28.843 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:35:28.843 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:35:28.843 7f514803eac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:35:38.092 7fda3e560c80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:35:38.092 7fda3e560c80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:35:38.092 7fda3e560c80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:35:38.096 7fda3e560c80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:35:38.096 7fda39ad92c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:35:38.097 7fda39ad92c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:35:38.097 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:35:38.097 7fda39ad97c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:35:38.098 7fda39ad97c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:35:38.098 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:35:38.098 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.099 7fda39ad97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:35:38.099 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.099 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.099 7fda39ad97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353338098190 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:35:38.099 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.100 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:38.101 7fda39ad97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.102 7fda39ad9e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:35:38.102 7fda39ad9e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:35:38.103 7fda39ad9c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:35:38.104 7fda39a89540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.104 7fda39a89540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:35:38.104 7fda39a89540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.104 7fda39a89540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.104 7fda39a89540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:38.104 7fda39a89540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.172 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:35:38.181 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:35:38.182 7fda3bbf1040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.182 7fda3bbf1040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:35:38.183 7fda3bbf1040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.183 7fda3bbf1040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.185 7fda3bbf1040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:35:38.185 7fda3bbf1040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.189 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:38.191 7fda39a897c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.192 7fda39a897c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:35:38.193 7fda39a897c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.194 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:38.195 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:38.196 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:38.196 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:38.197 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:35:38.197 7fda39a897c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.197 7fda39a897c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:38.198 7fda39a897c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.198 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:38.210 7fda3bbf1d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.210 7fda3bbf1d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:38.211 7fda3bbf1d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.213 7fda3bbf1d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.213 7fda3bbf1d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:38.213 7fda3bbf1d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.241 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:35:38.242 7fda3bbf1c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.242 7fda3bbf1c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:35:38.242 7fda3bbf1c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.251 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:35:38.252 7fda3bbf1c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.252 7fda3bbf1c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:35:38.252 7fda3bbf1c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.253 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:35:38.253 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:35:38.253 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:35:38.253 7fda3bbf1f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:35:38.257 7fda3b0ae220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.257 7fda3b0ae220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:35:38.257 7fda3b0ae220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.257 7fda3b0ae220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.257 7fda3b0ae220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:35:38.257 7fda3b0ae220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.261 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:35:38.263 7fda3bbf1f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:35:38.264 7fda3b0ae9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.264 7fda3b0ae9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:35:38.264 7fda3b0ae9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.292 7fda39ad97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:35:38.293 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.294 7fda3b0ae900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:35:38.294 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:35:38.330 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:38.331 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:38.331 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:38.332 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:38.332 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:38.332 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:38.562 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:38.563 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:38.563 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:38.564 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:38.565 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:38.565 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:39.016 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:39.017 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:39.017 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:39.018 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:39.018 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:39.018 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:39.103 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:40.104 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:40.226 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:40.227 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:40.228 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:40.228 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:40.228 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:40.229 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:41.105 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:42.098 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:35:42.099 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:35:42.099 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:35:42.099 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:35:42.099 7fda3b0ae900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:35:42.100 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:35:42.100 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:35:42.100 7fda3b0ae900 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:35:42.100 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:35:42.100 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:35:42.100 7fda3b0ae900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:35:42.100 7fda393ed180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:35:42.101 7fda393ed180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:35:42.106 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:42.163 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:42.164 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:42.164 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:42.164 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:42.164 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:42.165 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:43.107 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:44.109 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:45.109 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:45.341 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:35:45.341 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:45.341 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:35:45.342 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:45.342 7fda39a89a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:35:45.342 7fda39a89a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:35:46.111 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:47.112 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:48.113 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:49.114 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:50.116 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:51.117 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:52.118 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:53.120 7fda39ad9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:35:53.182 7fda39a89a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:35:53.182 7fda39a89a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:35:53.183 7fda39a89a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:35:53.324 7fda39ad97c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:35:53.324 7fda3e560c80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:35:53.324 7fda3e560c80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:35:53.324 7fda39a899a0 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:35:53.325 7fda39a899a0 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:35:53.339 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:35:53.339 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:35:53.339 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:35:53.340 7fda3e560c80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:36:02.268 7f55e89d2980 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:36:02.268 7f55e89d2980 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:36:02.268 7f55e89d2980 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:36:02.273 7f55e89d2980 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:36:02.273 7f55e39882c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:36:02.274 7f55e39882c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:36:02.274 7f55e39887c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:36:02.274 7f55e39887c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:36:02.275 7f55e39887c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:36:02.275 7f55e39887c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:36:02.275 7f55e39887c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.275 7f55e39887c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:36:02.276 7f55e39887c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.276 7f55e39887c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.276 7f55e39887c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353362275338 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:36:02.276 7f55e39887c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.277 7f55e39887c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.277 7f55e39887c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:02.277 7f55e39887c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.277 7f55e39887c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.277 7f55e39887c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:02.277 7f55e39887c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.277 7f55e39887c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.278 7f55e39887c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:02.278 7f55e39887c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.278 7f55e39887c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.278 7f55e39887c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:02.278 7f55e39887c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.278 7f55e3988e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:36:02.279 7f55e3988e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:36:02.280 7f55e3988c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:36:02.281 7f55e3938540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.281 7f55e3938540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:36:02.281 7f55e3938540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.281 7f55e3938540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.281 7f55e3938540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:02.281 7f55e3938540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.343 7f55e39887c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:36:02.352 7f55e39887c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:36:02.353 7f55e5a99040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.353 7f55e5a99040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:02.353 7f55e5a99040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.353 7f55e5a99040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.354 7f55e5a99040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:02.354 7f55e5a99040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.355 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:02.356 7f55e39387c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.356 7f55e39387c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:02.357 7f55e39387c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.357 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:02.357 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:02.358 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:02.358 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:02.359 7f55e39887c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:36:02.359 7f55e39387c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.359 7f55e39387c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:02.359 7f55e39387c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.360 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:02.367 7f55e5a99c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.368 7f55e5a99c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:02.368 7f55e5a99c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.369 7f55e5a99c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.369 7f55e5a99c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:02.369 7f55e5a99c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.396 7f55e39887c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:36:02.397 7f55e5a99ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.397 7f55e5a99ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:02.397 7f55e5a99ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.400 7f55e39887c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:36:02.401 7f55e5a99ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.401 7f55e5a99ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:02.401 7f55e5a99ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.401 7f55e39887c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:36:02.401 7f55e39887c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:36:02.402 7f55e39887c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:36:02.402 7f55e5a99e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:36:02.402 7f55e4f620e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.403 7f55e4f620e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:02.403 7f55e4f620e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.403 7f55e4f620e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.403 7f55e4f620e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:02.403 7f55e4f620e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.407 7f55e39887c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:36:02.409 7f55e5a99e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:36:02.409 7f55e4f629a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.409 7f55e4f629a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:02.409 7f55e4f629a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.434 7f55e39887c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:36:02.436 7f55e4f38400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.436 7f55e4f38400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:36:02.436 7f55e4f38400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:02.472 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:02.473 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:02.473 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:02.474 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:02.474 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:02.474 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:02.771 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:02.772 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:02.772 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:02.772 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:02.772 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:02.773 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:03.248 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:03.248 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:03.249 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:03.249 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:03.249 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:03.249 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:03.280 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:04.281 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:04.452 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:04.453 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:04.454 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:04.454 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:04.454 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:04.455 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:05.283 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:06.275 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:36:06.277 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:36:06.277 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:36:06.277 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:36:06.277 7f55e4f627c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:36:06.277 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:36:06.277 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:36:06.278 7f55e4f627c0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:36:06.278 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:36:06.278 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:36:06.278 7f55e4f627c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:36:06.278 7f55e4f62c20 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:36:06.280 7f55e4f62c20 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:36:06.285 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:06.914 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:06.915 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:06.915 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:06.916 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:06.916 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:06.916 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:07.286 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:08.287 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:09.288 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:10.289 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:10.860 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:10.861 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:10.861 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:10.861 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:10.862 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:10.862 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:11.291 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:12.292 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:13.293 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:14.294 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:15.295 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:16.296 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:16.468 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:16.469 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:16.469 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:16.471 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:16.471 7f55e3938a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:16.471 7f55e3938a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:17.298 7f55e3988f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:17.375 7f55e3938a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:36:17.375 7f55e3938a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:36:17.376 7f55e3938a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:36:17.510 7f55e39887c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:36:17.510 7f55e89d2980 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:36:17.510 7f55e89d2980 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:36:17.510 7f55e3938860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:36:17.510 7f55e3938860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:36:17.523 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:36:17.524 7f55e89d2980 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:36:26.541 7f8582361ec0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:36:26.541 7f8582361ec0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:36:26.541 7f8582361ec0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:36:26.544 7f8582361ec0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:36:26.545 7f857d5902c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:36:26.545 7f857d5902c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:36:26.545 7f857d5907c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:36:26.549 7f857d5907c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:36:26.550 7f857d5907c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:36:26.550 7f857d5907c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:36:26.550 7f857d5907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.550 7f857d5907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:36:26.550 7f857d5907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.550 7f857d5907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.550 7f857d5907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353386549979 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:36:26.550 7f857d5907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.552 7f857d5907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.552 7f857d5907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:26.552 7f857d5907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.552 7f857d5907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.552 7f857d5907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:26.552 7f857d5907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.553 7f857d5907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.553 7f857d5907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:26.553 7f857d5907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.553 7f857d5907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.553 7f857d5907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:26.553 7f857d5907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.553 7f857d590f40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:36:26.554 7f857d590f40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:36:26.557 7f857d590e00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:36:26.557 7f857d5407c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.558 7f857d5407c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:36:26.558 7f857d5407c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.558 7f857d5407c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.558 7f857d5407c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:26.558 7f857d5407c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.644 7f857d5907c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:36:26.656 7f857d5907c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:36:26.658 7f857d51b0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.658 7f857d51b0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:26.658 7f857d51b0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.658 7f857d51b0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.659 7f857d51b0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:26.659 7f857d51b0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.661 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:26.661 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:26.662 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:26.663 7f857d5409a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.664 7f857d5409a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:26.664 7f857d5409a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.665 7f857d5907c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:36:26.665 7f857d5409a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.666 7f857d5409a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:26.666 7f857d5409a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.666 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:26.666 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:26.667 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:26.669 7f857d51b0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.669 7f857d51b0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:26.669 7f857d51b0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.678 7f857d51bd60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.678 7f857d51bd60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:26.678 7f857d51bd60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.707 7f857d5907c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:36:26.707 7f857d51bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.708 7f857d51bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:26.708 7f857d51bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.711 7f857d5907c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:36:26.711 7f857d51bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.712 7f857d51bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:26.712 7f857d51bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.712 7f857d5907c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:36:26.712 7f857d5907c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:36:26.712 7f857d5907c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:36:26.713 7f857d51bf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:36:26.713 7f857d415220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.713 7f857d415220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:26.713 7f857d415220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.713 7f857d415220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.713 7f857d415220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:26.714 7f857d415220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.718 7f857d5907c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:36:26.722 7f857d51bf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:36:26.723 7f857d415c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.723 7f857d415c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:26.723 7f857d415c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.745 7f857d5907c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:36:26.747 7f857eb51400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.747 7f857eb51400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:36:26.747 7f857eb51400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:26.798 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:26.799 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:26.799 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:26.799 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:26.800 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:26.800 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:26.969 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:26.970 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:26.970 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:26.971 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:26.971 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:26.972 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:27.306 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:27.307 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:27.307 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:27.308 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:27.308 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:27.309 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:27.554 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:27.778 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:27.779 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:27.779 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:27.779 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:27.779 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:27.780 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:28.556 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:28.676 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:28.677 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:28.677 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:28.678 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:28.678 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:28.679 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:29.557 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:30.125 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:30.126 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:30.127 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:30.127 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:30.127 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:30.128 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:30.550 7f857eb51400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:36:30.552 7f857eb51400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:36:30.552 7f857eb51400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:36:30.552 7f857eb51400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:36:30.552 7f857eb51400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:36:30.553 7f857eb51400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:36:30.553 7f857eb51400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:36:30.553 7f857eb51400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:36:30.553 7f857eb51400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:36:30.553 7f857eb51400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:36:30.553 7f857eb51400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:36:30.553 7f857eb51900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:36:30.554 7f857eb51900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:36:30.559 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:31.560 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:31.867 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:31.868 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:31.868 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:31.868 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:31.868 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:31.870 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:32.561 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:33.563 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:34.564 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:35.523 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:35.524 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:35.524 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:35.524 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:35.525 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:35.525 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:35.565 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:36.566 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:37.567 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:38.568 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:39.569 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:40.570 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:40.843 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:40.844 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:40.844 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:40.845 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:40.845 7f857d540c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:40.845 7f857d540c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:41.571 7f857d5402c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:41.649 7f857d540c20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:36:41.649 7f857d540c20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:36:41.650 7f857d540c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:36:41.848 7f857d5907c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:36:41.849 7f8582361ec0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:36:41.849 7f8582361ec0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:36:41.849 7f857d540860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:36:41.849 7f857d540860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:36:41.900 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:36:41.900 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:36:41.900 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:36:41.901 7f8582361ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:36:50.546 7f771ccaa680 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:36:50.546 7f771ccaa680 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:36:50.546 7f771ccaa680 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:36:50.552 7f771ccaa680 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:36:50.552 7f771821e2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:36:50.552 7f771821e2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:36:50.554 7f771821e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:36:50.556 7f771821e7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:36:50.557 7f771821e7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:36:50.558 7f771821e7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:36:50.558 7f771821e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.561 7f771821e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:36:50.561 7f771821e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.561 7f771821e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.561 7f771821e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353410557054 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.189:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:36:50.561 7f771821e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.562 7f771821e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.562 7f771821e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:50.562 7f771821e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:50.563 7f771821e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.564 7f771821ee00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:36:50.564 7f771821ee00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:36:50.565 7f771821ec20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:36:50.566 7f77181cd540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.566 7f77181cd540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:36:50.566 7f77181cd540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.566 7f77181cd540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.567 7f77181cd540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:50.567 7f77181cd540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.681 7f771821e7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:36:50.709 7f771821e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:36:50.710 7f7718173040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.711 7f7718173040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:50.711 7f7718173040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.711 7f7718173040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.711 7f7718173040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:50.711 7f7718173040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.713 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:50.714 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:50.714 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:50.715 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:50.715 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:50.717 7f77181cd7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.717 7f77181cd7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:50.717 7f77181cd7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.717 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:50.720 7f77181cd7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.721 7f77181cd7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:50.721 7f77181cd7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.725 7f771821e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:36:50.725 7f77181cd7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.725 7f77181cd7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:50.726 7f77181cd7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.726 7f77181cd7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.726 7f77181cd7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:50.726 7f77181cd7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.808 7f771821e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:36:50.809 7f7718173ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.809 7f7718173ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:36:50.809 7f7718173ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.812 7f771821e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:36:50.813 7f7718173ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.813 7f7718173ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:36:50.813 7f7718173ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.813 7f771821e7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:36:50.814 7f771821e7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:36:50.814 7f771821e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:36:50.814 7f7718173e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:36:50.815 7f771809b0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.815 7f771809b0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:50.815 7f771809b0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.815 7f771809b0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.815 7f771809b0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:50.815 7f771809b0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.819 7f771821e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:36:50.822 7f7718173e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:36:50.823 7f771809b860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.823 7f771809b860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:36:50.823 7f771809b860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.869 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:50.880 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:50.881 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:50.882 7f771821e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:36:50.887 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:50.887 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:50.889 7f77197f5540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.892 7f77197f5540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:36:50.892 7f77197f5540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:36:50.897 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:51.193 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:51.201 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:51.201 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:51.202 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:51.202 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:51.202 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:51.565 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:51.608 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:51.609 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:51.609 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:51.610 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:51.610 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:51.610 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:52.304 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:52.305 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:52.305 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:52.305 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:52.306 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:52.306 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:52.566 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:53.574 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:53.627 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:53.628 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:53.628 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:53.628 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:53.628 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:53.629 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:54.555 7f77197f5540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:36:54.556 7f77197f5540 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:36:54.557 7f77197f5540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:36:54.557 7f77197f5540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:36:54.557 7f77197f5540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:36:54.557 7f77197f5540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:36:54.557 7f77197f5540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:36:54.557 7f77197f5540 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:36:54.558 7f77197f5540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:36:54.558 7f77197f5540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:36:54.558 7f77197f5540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:36:54.558 7f77197f5a40 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:36:54.560 7f77197f5a40 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:36:54.576 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:55.577 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:55.805 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:36:55.806 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:55.806 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:36:55.807 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:55.807 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:36:55.808 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:36:56.578 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:57.579 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:58.579 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:36:59.581 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:00.582 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:01.583 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:01.735 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:01.736 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:01.736 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:01.737 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:01.737 7f77181cda40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:01.737 7f77181cda40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:02.584 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:03.586 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:04.587 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:05.588 7f771821ef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:05.656 7f77181cda40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:37:05.656 7f77181cda40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:37:05.657 7f77181cda40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:37:05.979 7f771821e7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:37:05.980 7f771ccaa680 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:37:05.980 7f771ccaa680 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:37:05.980 7f77181cd680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:37:05.980 7f77181cd680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:37:05.996 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:37:05.997 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:37:05.997 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:37:05.997 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:37:05.997 7f771ccaa680 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:37:15.373 7f17b1ad4600 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:37:15.373 7f17b1ad4600 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:37:15.373 7f17b1ad4600 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:37:15.380 7f17b1ad4600 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:37:15.380 7f17ad04c2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:37:15.380 7f17ad04c2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:37:15.381 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:37:15.381 7f17ad04c7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:37:15.384 7f17ad04c7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:37:15.385 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:37:15.385 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.385 7f17ad04c7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:37:15.385 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.385 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.385 7f17ad04c7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353435384826 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:37:15.395 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.396 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:15.397 7f17ad04c7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.398 7f17ad04ce00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:37:15.398 7f17ad04ce00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:37:15.399 7f17ad04cc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:37:15.400 7f17acffc540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.400 7f17acffc540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:37:15.400 7f17acffc540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.400 7f17acffc540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.400 7f17acffc540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:15.400 7f17acffc540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.463 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:37:15.472 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:37:15.473 7f17acf550e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.473 7f17acf550e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:37:15.473 7f17acf550e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.473 7f17acf550e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.473 7f17acf550e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:37:15.473 7f17acf550e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.475 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:15.476 7f17acffc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.476 7f17acffc7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:37:15.476 7f17acffc7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.477 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:15.477 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:15.478 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:37:15.478 7f17acffc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.479 7f17acffc7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:15.479 7f17acffc7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.479 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:15.479 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:15.480 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:15.488 7f17acf55d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.488 7f17acf55d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:15.488 7f17acf55d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.490 7f17acf55d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.490 7f17acf55d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:15.490 7f17acf55d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.516 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:37:15.517 7f17acf55c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.517 7f17acf55c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:37:15.517 7f17acf55c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.522 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:37:15.522 7f17acf55c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.522 7f17acf55c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:15.522 7f17acf55c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.523 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:37:15.523 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:37:15.523 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:37:15.524 7f17acf55f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:37:15.524 7f17aced0220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.525 7f17aced0220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:37:15.525 7f17aced0220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.525 7f17aced0220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.525 7f17aced0220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:37:15.525 7f17aced0220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.529 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:37:15.531 7f17acf55f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:37:15.531 7f17aced09a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.531 7f17aced09a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:37:15.532 7f17aced09a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.576 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:15.577 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:15.577 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:15.578 7f17ad04c7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:37:15.579 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:15.579 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:15.582 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.582 7f17ae6e3400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:37:15.583 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:15.583 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:15.782 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:15.783 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:15.783 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:15.784 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:15.784 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:15.784 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:16.105 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:16.106 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:16.106 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:16.107 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:16.107 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:16.107 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:16.399 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:16.610 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:16.611 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:16.611 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:16.612 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:16.612 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:16.612 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:17.400 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:17.674 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:17.675 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:17.675 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:17.676 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:17.676 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:17.677 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:18.401 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:19.293 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:19.294 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:19.294 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:19.294 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:19.294 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:19.295 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:19.382 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:37:19.384 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:37:19.384 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:37:19.384 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:37:19.384 7f17ae6e3400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:37:19.384 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:37:19.384 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:37:19.384 7f17ae6e3400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:37:19.385 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:37:19.385 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:37:19.385 7f17ae6e3400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:37:19.385 7f17ae6e3900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:37:19.386 7f17ae6e3900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:37:19.403 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:20.404 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:21.405 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:22.406 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:22.459 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:22.460 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:22.460 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:22.462 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:22.462 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:22.463 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:23.408 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:24.408 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:25.409 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:26.411 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:27.412 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:28.413 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:29.414 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:30.059 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:30.060 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:30.062 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:30.063 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:30.063 7f17acffcc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:30.063 7f17acffcc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:30.415 7f17ad04cf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:30.465 7f17acffcc20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:37:30.466 7f17acffcc20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:37:30.466 7f17acffcc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:37:30.637 7f17ad04c7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:37:30.638 7f17b1ad4600 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:37:30.638 7f17b1ad4600 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:37:30.638 7f17acffc680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:37:30.638 7f17acffc680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:37:30.653 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:37:30.654 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:37:30.655 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:37:30.655 7f17b1ad4600 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:37:39.636 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:37:39.636 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:37:39.636 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:37:39.639 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:37:39.640 7ff9f8ae12c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:37:39.640 7ff9f8ae12c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:37:39.640 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:37:39.640 7ff9f8ae17c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:37:39.640 7ff9f8ae17c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:37:39.641 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:37:39.641 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.641 7ff9f8ae17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:37:39.641 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.641 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.641 7ff9f8ae17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353459640921 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.189:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:37:39.641 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.642 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.642 7ff9f8ae17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:37:39.642 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.642 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.642 7ff9f8ae17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:37:39.642 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.642 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.643 7ff9f8ae17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:37:39.643 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.643 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.643 7ff9f8ae17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:39.643 7ff9f8ae17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.643 7ff9f8ae1e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:37:39.643 7ff9f8ae1e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:37:39.644 7ff9f8ae1c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:37:39.645 7ff9f8a91540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.645 7ff9f8a91540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:37:39.645 7ff9f8a91540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.645 7ff9f8a91540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.645 7ff9f8a91540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:39.645 7ff9f8a91540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.713 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:37:39.722 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:37:39.723 7ff9f8a70040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.723 7ff9f8a70040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:37:39.723 7ff9f8a70040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.724 7ff9f8a70040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.724 7ff9f8a70040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:37:39.724 7ff9f8a70040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.725 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:39.726 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:39.726 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:39.727 7ff9f8a917c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.727 7ff9f8a917c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:37:39.727 7ff9f8a917c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.728 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:39.728 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:39.729 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:37:39.729 7ff9f8a700e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.730 7ff9f8a700e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:39.730 7ff9f8a700e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.730 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:39.731 7ff9f8a70860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.731 7ff9f8a70860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:39.732 7ff9f8a70860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.740 7ff9f8a70d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.740 7ff9f8a70d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:39.740 7ff9f8a70d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.780 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:37:39.782 7ff9f8a70c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.783 7ff9f8a70c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:37:39.783 7ff9f8a70c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.788 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:37:39.788 7ff9f8a70c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.789 7ff9f8a70c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:37:39.789 7ff9f8a70c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.793 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:37:39.793 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:37:39.794 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:37:39.794 7ff9f8a70f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:37:39.795 7ff9f83ad220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.795 7ff9f83ad220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:37:39.795 7ff9f83ad220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.795 7ff9f83ad220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.795 7ff9f83ad220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:37:39.795 7ff9f83ad220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.798 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:37:39.807 7ff9f8a70f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:37:39.807 7ff9f83adea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.808 7ff9f83adea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:37:39.808 7ff9f83adea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.808 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:39.809 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:39.809 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:39.810 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:39.810 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:39.811 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:39.830 7ff9f8ae17c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:37:39.832 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.832 7ff9f83ad900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:37:39.833 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:37:39.977 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:39.978 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:39.978 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:39.978 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:39.978 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:39.980 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:40.396 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:40.397 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:40.397 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:40.398 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:40.398 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:40.398 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:40.644 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:40.961 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:40.961 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:40.962 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:40.962 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:40.962 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:40.963 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:41.645 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:42.338 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:42.340 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:42.340 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:42.340 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:42.340 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:42.341 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:42.646 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:43.641 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:37:43.642 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:37:43.642 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:37:43.643 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:37:43.643 7ff9f83ad900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:37:43.643 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:37:43.643 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:37:43.643 7ff9f83ad900 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:37:43.643 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:37:43.644 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:37:43.644 7ff9f83ad900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:37:43.644 7ff9fabd7360 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:37:43.645 7ff9fabd7360 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:37:43.648 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:44.135 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:44.136 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:44.136 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:44.137 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:44.137 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:44.137 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:44.649 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:45.650 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:46.651 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:46.793 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:46.793 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:46.794 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:46.795 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:46.795 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:46.795 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:47.652 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:48.653 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:49.655 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:50.656 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:51.658 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:52.659 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:53.660 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:53.711 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:37:53.712 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:53.712 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:37:53.712 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:53.712 7ff9f8a91a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:37:53.713 7ff9f8a91a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:37:54.662 7ff9f8ae1f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:37:54.717 7ff9f8a91a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:37:54.717 7ff9f8a91a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:37:54.718 7ff9f8a91a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:37:54.884 7ff9f8ae17c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:37:54.885 7ff9fd56ae00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:37:54.885 7ff9fd56ae00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:37:54.885 7ff9f8a919a0 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:37:54.885 7ff9f8a919a0 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:37:54.899 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:37:54.899 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:37:54.899 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:37:54.900 7ff9fd56ae00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:38:03.808 7f6c6a625d00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:38:03.808 7f6c6a625d00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:38:03.808 7f6c6a625d00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:38:03.812 7f6c6a625d00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:38:03.813 7f6c655c52c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:38:03.813 7f6c655c52c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:38:03.813 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:38:03.813 7f6c655c57c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:38:03.814 7f6c655c57c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:38:03.814 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:38:03.814 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.814 7f6c655c57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:38:03.814 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.814 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.814 7f6c655c57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353483814233 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.189:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:38:03.814 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.815 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.816 7f6c655c57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:03.816 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.816 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.816 7f6c655c57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:03.816 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.816 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.816 7f6c655c57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:03.817 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.817 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.817 7f6c655c57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:03.817 7f6c655c57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.817 7f6c655c5e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:38:03.817 7f6c655c5e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:38:03.818 7f6c655c5c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:38:03.819 7f6c65575540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.819 7f6c65575540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:38:03.819 7f6c65575540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.819 7f6c65575540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.819 7f6c65575540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:03.819 7f6c65575540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.891 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:38:03.901 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:38:03.901 7f6c65557040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.902 7f6c65557040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:03.902 7f6c65557040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.902 7f6c65557040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.902 7f6c65557040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:03.902 7f6c65557040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.904 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:03.905 7f6c655757c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.905 7f6c655757c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:03.905 7f6c655757c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.906 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:03.906 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:03.908 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:38:03.908 7f6c655757c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.908 7f6c655757c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:03.908 7f6c655757c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.908 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:03.908 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:03.909 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:03.910 7f6c65557860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.910 7f6c65557860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:03.910 7f6c65557860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.918 7f6c65557d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.918 7f6c65557d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:03.918 7f6c65557d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.952 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:38:03.953 7f6c65557c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.954 7f6c65557c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:03.954 7f6c65557c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.957 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:38:03.958 7f6c65557c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.958 7f6c65557c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:03.958 7f6c65557c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.958 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:38:03.959 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:38:03.959 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:38:03.959 7f6c65557f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:38:03.960 7f6c64e8a220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.961 7f6c64e8a220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:03.961 7f6c64e8a220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.961 7f6c64e8a220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.961 7f6c64e8a220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:03.961 7f6c64e8a220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.964 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:38:03.966 7f6c65557f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:38:03.966 7f6c64e8a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.966 7f6c64e8a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:03.966 7f6c64e8a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.989 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:03.990 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:03.990 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:03.991 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:03.991 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:03.992 7f6c655c57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:38:03.992 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:03.994 7f6c676cf400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:03.994 7f6c676cf400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:38:03.994 7f6c676cf400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:04.186 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:04.187 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:04.187 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:04.188 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:04.188 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:04.188 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:04.455 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:04.456 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:04.456 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:04.457 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:04.457 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:04.457 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:04.818 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:04.973 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:04.976 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:04.976 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:04.977 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:04.977 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:04.977 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:05.820 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:05.900 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:05.900 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:05.900 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:05.901 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:05.901 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:05.901 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:06.821 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:07.814 7f6c676cf400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:38:07.815 7f6c676cf400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:38:07.816 7f6c676cf400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:38:07.816 7f6c676cf400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:38:07.816 7f6c676cf400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:38:07.816 7f6c676cf400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:38:07.816 7f6c676cf400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:38:07.816 7f6c676cf400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:38:07.816 7f6c676cf400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:38:07.817 7f6c676cf400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:38:07.817 7f6c676cf400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:38:07.817 7f6c676cf900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:38:07.818 7f6c676cf900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:38:07.822 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:07.887 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:07.888 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:07.888 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:07.889 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:07.889 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:07.889 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:08.823 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:09.824 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:10.826 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:11.630 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:11.631 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:11.631 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:11.632 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:11.632 7f6c65575a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:11.632 7f6c65575a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:11.827 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:12.829 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:13.830 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:14.832 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:15.833 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:16.834 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:17.835 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:18.836 7f6c655c5f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:18.865 7f6c65575a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:38:18.865 7f6c65575a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:38:18.866 7f6c65575a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:38:19.070 7f6c655c57c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:38:19.071 7f6c6a625d00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:38:19.071 7f6c6a625d00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:38:19.071 7f6c65575680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:38:19.071 7f6c65575680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:38:19.086 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:38:19.086 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:38:19.086 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:38:19.086 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:38:19.086 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:38:19.086 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:38:19.086 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:38:19.086 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:38:19.087 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:38:19.087 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:38:19.087 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:38:19.087 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:38:19.087 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:38:19.087 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:38:19.087 7f6c6a625d00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:38:27.962 7ff1f942bd00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:38:27.962 7ff1f942bd00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:38:27.962 7ff1f942bd00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:38:27.965 7ff1f942bd00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:38:27.965 7ff1f49ab2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:38:27.966 7ff1f49ab2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:38:27.966 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:38:27.966 7ff1f49ab7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:38:27.966 7ff1f49ab7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:38:27.967 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:38:27.967 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.967 7ff1f49ab7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:38:27.967 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.967 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.967 7ff1f49ab7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353507966822 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.50:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:38:27.967 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.968 7ff1f49ab7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:27.969 7ff1f49ab7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.969 7ff1f49abe00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:38:27.969 7ff1f49abe00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:38:27.970 7ff1f49abc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:38:27.970 7ff1f495b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.970 7ff1f495b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:38:27.971 7ff1f495b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.971 7ff1f495b540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:27.971 7ff1f495b540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:27.971 7ff1f495b540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.023 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:38:28.030 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:38:28.031 7ff1f48b7040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.032 7ff1f48b7040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:28.032 7ff1f48b7040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.032 7ff1f48b7040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.032 7ff1f48b7040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:28.032 7ff1f48b7040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.033 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:28.034 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.034 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:28.035 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.036 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:28.037 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:38:28.037 7ff1f495b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.038 7ff1f495b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:28.038 7ff1f495b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.038 7ff1f495b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.038 7ff1f495b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:28.038 7ff1f495b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.039 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.049 7ff1f48b7c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.050 7ff1f48b7c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:28.050 7ff1f48b7c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.051 7ff1f48b7c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.051 7ff1f48b7c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:28.052 7ff1f48b7c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.074 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:38:28.075 7ff1f48b7ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.076 7ff1f48b7ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:28.076 7ff1f48b7ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.079 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:38:28.079 7ff1f48b7ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.079 7ff1f48b7ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:28.079 7ff1f48b7ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.080 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:38:28.080 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:38:28.080 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:38:28.080 7ff1f48b7e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:38:28.081 7ff1f482e0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.081 7ff1f482e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:28.081 7ff1f482e0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.081 7ff1f482e0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.081 7ff1f482e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:28.081 7ff1f482e0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.085 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:38:28.087 7ff1f48b7e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:38:28.087 7ff1f482e9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.088 7ff1f482e9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:28.088 7ff1f482e9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.109 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:28.109 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.109 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:28.110 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.110 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:28.111 7ff1f49ab7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:38:28.111 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.112 7ff1f6041540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.113 7ff1f6041540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:38:28.113 7ff1f6041540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:28.195 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:28.195 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.195 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:28.196 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.196 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:28.196 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.394 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:28.395 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.395 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:28.395 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.396 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:28.396 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.635 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:28.636 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.636 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:28.637 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.637 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:28.637 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:28.970 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:29.163 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:29.164 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:29.164 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:29.164 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:29.165 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:29.165 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:29.972 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:30.146 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:30.147 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:30.147 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:30.147 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:30.148 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:30.148 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:30.972 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:31.967 7ff1f6041180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:38:31.968 7ff1f6041180 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:38:31.968 7ff1f6041180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:38:31.969 7ff1f6041180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:38:31.969 7ff1f6041180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:38:31.969 7ff1f6041180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:38:31.969 7ff1f6041180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:38:31.969 7ff1f6041180 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:38:31.970 7ff1f6041180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:38:31.970 7ff1f6041180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:38:31.970 7ff1f6041180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:38:31.970 7ff1f6041540 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:38:31.971 7ff1f6041540 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:38:31.973 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:32.545 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:32.546 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:32.546 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:32.547 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:32.547 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:32.548 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:32.974 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:33.975 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:34.976 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:35.977 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:36.978 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:37.000 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:37.001 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:37.001 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:37.001 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:37.001 7ff1f495ba40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:37.002 7ff1f495ba40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:37.980 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:38.981 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:39.982 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:40.983 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:41.984 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:42.986 7ff1f49abf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:43.031 7ff1f495ba40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:38:43.031 7ff1f495ba40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:38:43.031 7ff1f495ba40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:38:43.118 7ff1f49ab7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:38:43.119 7ff1f942bd00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:38:43.119 7ff1f942bd00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:38:43.119 7ff1f495b860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:38:43.119 7ff1f495b860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:38:43.133 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:38:43.134 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:38:43.134 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:38:43.134 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:38:43.134 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:38:43.134 7ff1f942bd00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:38:52.434 7fcbc04ab880 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:38:52.434 7fcbc04ab880 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:38:52.434 7fcbc04ab880 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:38:52.437 7fcbc04ab880 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:38:52.438 7fcbbba272c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:38:52.438 7fcbbba272c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:38:52.438 7fcbbba277c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:38:52.438 7fcbbba277c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:38:52.439 7fcbbba277c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:38:52.439 7fcbbba277c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:38:52.439 7fcbbba277c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.439 7fcbbba277c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:38:52.439 7fcbbba277c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.439 7fcbbba277c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.439 7fcbbba277c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353532439035 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:38:52.439 7fcbbba277c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.440 7fcbbba27900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:52.441 7fcbbba27900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.442 7fcbbba27f40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:38:52.442 7fcbbba27f40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:38:52.443 7fcbbba27e00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:38:52.444 7fcbbb9d77c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.444 7fcbbb9d77c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:38:52.444 7fcbbb9d77c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.444 7fcbbb9d77c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.444 7fcbbb9d77c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:52.444 7fcbbb9d77c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.511 7fcbbba27900 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:38:52.519 7fcbbba27900 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:38:52.520 7fcbbdb330e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.521 7fcbbdb330e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:52.521 7fcbbdb330e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.521 7fcbbdb330e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.521 7fcbbdb330e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:52.521 7fcbbdb330e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.522 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:52.522 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:52.522 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:52.526 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:52.526 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:52.527 7fcbbb9d79a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.527 7fcbbb9d79a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:52.527 7fcbbb9d79a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.528 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:52.528 7fcbbba27900 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:38:52.528 7fcbbb9d79a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.528 7fcbbb9d79a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:52.528 7fcbbb9d79a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.537 7fcbbdb33d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.537 7fcbbdb33d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:52.537 7fcbbdb33d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.539 7fcbbdb33d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.539 7fcbbdb33d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:52.539 7fcbbdb33d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.572 7fcbbba27900 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:38:52.586 7fcbbdb33c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.586 7fcbbdb33c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:38:52.586 7fcbbdb33c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.590 7fcbbba27900 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:38:52.590 7fcbbdb33c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.591 7fcbbdb33c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:38:52.591 7fcbbdb33c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.591 7fcbbba27900 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:38:52.591 7fcbbba27900 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:38:52.592 7fcbbba27900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:38:52.592 7fcbbdb33f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:38:52.592 7fcbbd000220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.593 7fcbbd000220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:52.593 7fcbbd000220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.593 7fcbbd000220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.593 7fcbbd000220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:52.593 7fcbbd000220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.596 7fcbbba27900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:38:52.597 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:52.598 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:52.598 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:52.599 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:52.599 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:52.600 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:52.602 7fcbbdb33f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:38:52.602 7fcbbd000c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.603 7fcbbd000c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:38:52.603 7fcbbd000c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.626 7fcbbba27900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:38:52.627 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.628 7fcbbcfd7400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:38:52.628 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:38:52.747 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:52.748 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:52.748 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:52.749 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:52.749 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:52.750 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:53.117 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:53.117 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:53.117 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:53.118 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:53.118 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:53.119 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:53.444 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:53.934 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:53.935 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:53.935 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:53.935 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:53.935 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:53.936 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:54.445 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:55.013 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:55.013 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:55.013 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:55.014 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:55.014 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:55.014 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:55.446 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:56.439 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:38:56.441 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:38:56.441 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:38:56.441 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:38:56.441 7fcbbcfd7400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:38:56.441 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:38:56.441 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:38:56.442 7fcbbcfd7400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:38:56.442 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:38:56.442 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:38:56.442 7fcbbcfd7400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:38:56.442 7fcbbcfd7900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:38:56.443 7fcbbcfd7900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:38:56.447 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:57.448 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:57.468 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:38:57.469 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:57.469 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:38:57.470 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:57.470 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:38:57.471 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:38:58.450 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:38:59.454 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:00.456 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:01.166 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:01.167 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:01.167 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:01.167 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:01.167 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:01.168 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:01.457 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:02.458 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:03.460 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:04.461 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:05.462 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:06.463 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:07.213 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:07.214 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:07.214 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:07.214 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:07.215 7fcbbb9d7c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:07.215 7fcbbb9d7c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:07.465 7fcbbb9d72c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:07.517 7fcbbb9d7c20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:39:07.517 7fcbbb9d7c20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:39:07.518 7fcbbb9d7c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:39:07.716 7fcbbba27900 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:39:07.716 7fcbc04ab880 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:39:07.716 7fcbc04ab880 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:39:07.716 7fcbbb9d7860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:39:07.716 7fcbbb9d7860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:39:07.738 7fcbc04ab880 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:39:16.818 7f4915cc1d40 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:39:16.819 7f4915cc1d40 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:39:16.819 7f4915cc1d40 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:39:16.822 7f4915cc1d40 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:39:16.823 7f491124a2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:39:16.823 7f491124a2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:39:16.823 7f491124a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:39:16.824 7f491124a7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:39:16.824 7f491124a7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:39:16.824 7f491124a7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:39:16.824 7f491124a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.825 7f491124a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:39:16.825 7f491124a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.825 7f491124a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.825 7f491124a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353556824396 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:39:16.825 7f491124a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.826 7f491124a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.827 7f491124a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:16.827 7f491124a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.827 7f491124ae00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:39:16.827 7f491124ae00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:39:16.828 7f491124ac20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:39:16.829 7f49111fa540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.829 7f49111fa540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:39:16.829 7f49111fa540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.829 7f49111fa540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.829 7f49111fa540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:16.829 7f49111fa540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.883 7f491124a7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:39:16.890 7f491124a7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:39:16.891 7f4911197040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.892 7f4911197040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:16.892 7f4911197040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.892 7f4911197040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.892 7f4911197040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:16.892 7f4911197040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.893 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:16.894 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:16.894 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:16.895 7f49111fa7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.895 7f49111fa7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:16.895 7f49111fa7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.896 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:16.896 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:16.897 7f491124a7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:39:16.897 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:16.897 7f49111fa7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.898 7f49111fa7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:16.898 7f49111fa7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.899 7f4911197040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.899 7f4911197040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:16.899 7f4911197040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.906 7f4911197d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.906 7f4911197d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:16.906 7f4911197d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.937 7f491124a7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:39:16.938 7f4911197c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.938 7f4911197c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:16.938 7f4911197c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.942 7f491124a7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:39:16.942 7f4911197c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.942 7f4911197c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:16.942 7f4911197c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.943 7f491124a7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:39:16.943 7f491124a7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:39:16.943 7f491124a7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:39:16.943 7f4911197f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:39:16.944 7f49110ce220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.944 7f49110ce220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:39:16.944 7f49110ce220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.944 7f49110ce220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.944 7f49110ce220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:39:16.944 7f49110ce220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.948 7f491124a7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:39:16.959 7f4911197f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:39:16.959 7f49110ceae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.959 7f49110ceae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:39:16.959 7f49110ceae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.971 7f491124a7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:39:16.973 7f49128dd400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.973 7f49128dd400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:39:16.973 7f49128dd400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:16.976 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:16.977 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:16.977 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:16.977 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:16.977 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:16.977 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.112 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:17.113 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.113 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:17.113 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.114 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:17.114 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.362 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:17.363 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.363 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:17.363 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.363 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:17.364 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.797 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:17.798 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.798 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:17.799 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.799 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:17.799 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:17.829 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:18.830 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:18.986 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:18.986 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:18.987 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:18.988 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:18.988 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:18.988 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:19.831 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:20.824 7f49128dd400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:39:20.825 7f49128dd400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:39:20.825 7f49128dd400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:39:20.826 7f49128dd400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:39:20.826 7f49128dd400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:39:20.826 7f49128dd400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:39:20.826 7f49128dd400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:39:20.826 7f49128dd400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:39:20.826 7f49128dd400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:39:20.827 7f49128dd400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:39:20.827 7f49128dd400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:39:20.827 7f49128dd900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:39:20.828 7f49128dd900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:39:20.835 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:21.836 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:22.055 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:22.056 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:22.056 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:22.057 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:22.057 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:22.057 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:22.837 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:23.838 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:24.839 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:25.840 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:26.601 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:26.603 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:26.603 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:26.604 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:26.604 7f49111faa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:26.604 7f49111faa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:26.842 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:27.844 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:28.845 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:29.847 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:30.848 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:31.849 7f491124af40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:31.929 7f49111faa40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:39:31.929 7f49111faa40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:39:31.929 7f49111faa40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:39:31.987 7f491124a7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:39:31.987 7f4915cc1d40 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:39:31.988 7f4915cc1d40 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:39:31.988 7f49111fa680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:39:31.988 7f49111fa680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:39:32.007 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:39:32.008 7f4915cc1d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:39:40.892 7f75063ebbc0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:39:40.892 7f75063ebbc0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:39:40.892 7f75063ebbc0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:39:40.896 7f75063ebbc0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:39:40.896 7f75019642c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:39:40.897 7f75019642c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:39:40.897 7f75019647c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:39:40.897 7f75019647c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:39:40.898 7f75019647c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:39:40.898 7f75019647c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:39:40.898 7f75019647c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.898 7f75019647c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:39:40.898 7f75019647c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.899 7f75019647c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.899 7f75019647c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353580898158 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:39:40.899 7f75019647c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.900 7f75019647c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.901 7f75019647c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:40.901 7f75019647c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.901 7f7501964f40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:39:40.901 7f7501964f40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:39:40.902 7f7501964e00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:39:40.903 7f75019147c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.903 7f75019147c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:39:40.903 7f75019147c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.903 7f75019147c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.903 7f75019147c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:40.903 7f75019147c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.972 7f75019647c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:39:40.980 7f75019647c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:39:40.981 7f7503a760e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.982 7f7503a760e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:40.982 7f7503a760e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.982 7f7503a760e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.982 7f7503a760e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:40.982 7f7503a760e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.983 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:40.984 7f75019149a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.985 7f75019149a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:40.985 7f75019149a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.985 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:40.985 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:40.986 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:40.987 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:40.988 7f75019647c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:39:40.988 7f75019149a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.988 7f75019149a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:40.988 7f75019149a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.988 7f75019149a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.988 7f75019149a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:40.988 7f75019149a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.989 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:40.996 7f7503a76d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:40.996 7f7503a76d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:40.997 7f7503a76d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.024 7f75019647c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:39:41.027 7f7503a76c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.027 7f7503a76c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:39:41.027 7f7503a76c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.030 7f75019647c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:39:41.030 7f7503a76c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.031 7f7503a76c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:39:41.031 7f7503a76c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.031 7f75019647c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:39:41.031 7f75019647c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:39:41.031 7f75019647c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:39:41.031 7f7503a76f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:39:41.032 7f7502f3a220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.032 7f7502f3a220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:39:41.032 7f7502f3a220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.032 7f7502f3a220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.033 7f7502f3a220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:39:41.033 7f7502f3a220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.036 7f75019647c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:39:41.039 7f7503a76f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:39:41.040 7f7502f3a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.040 7f7502f3a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:39:41.040 7f7502f3a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.065 7f75019647c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:39:41.067 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.068 7f7502f3a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:39:41.068 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:39:41.096 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:41.096 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.097 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:41.097 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.097 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:41.098 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.232 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:41.232 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.232 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:41.233 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.233 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:41.233 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.483 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:41.484 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.484 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:41.484 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.484 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:41.485 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:41.903 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:42.126 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:42.127 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:42.127 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:42.127 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:42.127 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:42.128 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:42.903 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:43.092 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:43.093 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:43.094 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:43.094 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:43.094 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:43.095 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:43.905 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:44.561 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:44.562 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:44.563 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:44.563 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:44.564 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:44.564 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:44.898 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:39:44.899 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:39:44.899 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:39:44.899 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:39:44.899 7f7502f3a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:39:44.899 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:39:44.899 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:39:44.900 7f7502f3a7c0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:39:44.900 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:39:44.900 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:39:44.900 7f7502f3a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:39:44.900 7f750127c180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:39:44.901 7f750127c180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:39:44.906 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:45.907 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:46.909 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:46.947 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:46.948 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:46.948 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:46.948 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:46.948 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:46.949 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:47.910 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:48.911 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:49.912 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:50.913 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:51.915 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:52.311 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:39:52.311 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:52.311 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:39:52.312 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:52.312 7f7501914c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:39:52.312 7f7501914c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:39:52.916 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:53.918 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:54.919 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:55.921 7f75019142c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:39:55.926 7f7501914c20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:39:55.926 7f7501914c20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:39:55.927 7f7501914c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:39:56.157 7f75019647c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:39:56.158 7f75063ebbc0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:39:56.158 7f75063ebbc0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:39:56.158 7f7501914b80 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:39:56.158 7f7501914b80 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:39:56.180 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:39:56.181 7f75063ebbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:40:05.487 7fe3113f5d40 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:40:05.487 7fe3113f5d40 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:40:05.487 7fe3113f5d40 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:40:05.490 7fe3113f5d40 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:40:05.491 7fe30c3a02c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:40:05.491 7fe30c3a02c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:40:05.491 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:40:05.492 7fe30c3a07c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:40:05.492 7fe30c3a07c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:40:05.492 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:40:05.492 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.492 7fe30c3a07c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:40:05.492 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.493 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.493 7fe30c3a07c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353605492451 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:40:05.493 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.493 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.493 7fe30c3a07c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:05.493 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:05.494 7fe30c3a07c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.494 7fe30c3a0f40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:40:05.495 7fe30c3a0f40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:40:05.496 7fe30c3a0e00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:40:05.496 7fe30c3507c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.496 7fe30c3507c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:40:05.496 7fe30c3507c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.496 7fe30c3507c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.496 7fe30c3507c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:05.496 7fe30c3507c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.549 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:40:05.557 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:40:05.557 7fe30e4ac0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.558 7fe30e4ac0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:05.558 7fe30e4ac0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.558 7fe30e4ac0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.558 7fe30e4ac0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:05.558 7fe30e4ac0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.559 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:05.560 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:05.560 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:05.561 7fe30c3509a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.561 7fe30c3509a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:05.561 7fe30c3509a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.561 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:05.561 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:05.562 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:05.563 7fe30c3509a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.564 7fe30c3509a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:05.564 7fe30c3509a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.565 7fe30c3509a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.565 7fe30c3509a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:05.565 7fe30c3509a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.572 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:40:05.573 7fe30c3509a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.573 7fe30c3509a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:05.573 7fe30c3509a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.615 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:40:05.616 7fe30e4acc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.616 7fe30e4acc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:05.617 7fe30e4acc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.619 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:40:05.620 7fe30e4acc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.620 7fe30e4acc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:05.620 7fe30e4acc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.620 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:40:05.620 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:40:05.620 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:40:05.621 7fe30e4acf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:40:05.621 7fe30d978220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.622 7fe30d978220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:05.622 7fe30d978220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.622 7fe30d978220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.622 7fe30d978220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:05.622 7fe30d978220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.625 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:40:05.627 7fe30e4acf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:40:05.627 7fe30d9789a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.628 7fe30d9789a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:05.628 7fe30d9789a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.649 7fe30c3a07c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:40:05.651 7fe30d952400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.651 7fe30d952400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:40:05.651 7fe30d952400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:05.681 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:05.681 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:05.682 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:05.682 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:05.682 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:05.682 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:06.013 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:06.013 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:06.014 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:06.014 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:06.014 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:06.015 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:06.496 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:06.668 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:06.668 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:06.669 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:06.669 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:06.669 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:06.670 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:07.497 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:07.843 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:07.844 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:07.844 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:07.845 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:07.845 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:07.845 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:08.498 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:09.493 7fe30d952400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:40:09.494 7fe30d952400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:40:09.494 7fe30d952400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:40:09.494 7fe30d952400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:40:09.494 7fe30d952400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:40:09.494 7fe30d952400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:40:09.494 7fe30d952400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:40:09.494 7fe30d952400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:40:09.495 7fe30d952400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:40:09.495 7fe30d952400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:40:09.495 7fe30d952400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:40:09.495 7fe30d952900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:40:09.496 7fe30d952900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:40:09.499 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:10.450 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:10.451 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:10.451 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:10.452 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:10.452 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:10.453 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:10.500 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:11.502 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:12.503 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:13.504 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:13.856 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:13.856 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:13.857 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:13.857 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:13.857 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:13.858 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:14.505 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:15.507 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:16.508 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:17.510 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:18.511 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:19.512 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:19.816 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:19.817 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:19.817 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:19.818 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:19.818 7fe30c350c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:19.819 7fe30c350c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:20.514 7fe30c3502c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:20.522 7fe30c350c20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:40:20.523 7fe30c350c20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:40:20.523 7fe30c350c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:40:20.748 7fe30c3a07c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:40:20.748 7fe3113f5d40 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:40:20.749 7fe3113f5d40 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:40:20.749 7fe30c350860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:40:20.749 7fe30c350860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:40:20.762 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:40:20.763 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:40:20.764 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:40:20.764 7fe3113f5d40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:40:29.546 7fde18ca3c40 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:40:29.546 7fde18ca3c40 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:40:29.547 7fde18ca3c40 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:40:29.550 7fde18ca3c40 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:40:29.551 7fde1420f2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:40:29.551 7fde1420f2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:40:29.551 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:40:29.551 7fde1420f7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:40:29.552 7fde1420f7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:40:29.552 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:40:29.552 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.552 7fde1420f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:40:29.552 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.552 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.553 7fde1420f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353629552126 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:40:29.553 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:29.554 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.555 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.555 7fde1420f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:29.555 7fde1420f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.555 7fde1420fe00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:40:29.555 7fde1420fe00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:40:29.557 7fde1420fc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:40:29.557 7fde141bf540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.557 7fde141bf540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:40:29.557 7fde141bf540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.558 7fde141bf540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.558 7fde141bf540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:29.558 7fde141bf540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.628 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:40:29.637 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:40:29.639 7fde141460e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.639 7fde141460e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:29.639 7fde141460e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.639 7fde141460e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.639 7fde141460e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:29.639 7fde141460e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.642 7fde141bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.643 7fde141bf7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:29.643 7fde141bf7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.643 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:29.644 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:40:29.645 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:29.645 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:29.645 7fde141bf7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.645 7fde141bf7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:29.645 7fde141bf7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.646 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:29.646 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:29.653 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:29.654 7fde14146d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.655 7fde14146d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:29.655 7fde14146d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.656 7fde14146d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.656 7fde14146d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:29.656 7fde14146d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.684 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:40:29.685 7fde14146c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.686 7fde14146c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:29.686 7fde14146c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.689 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:40:29.690 7fde14146c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.690 7fde14146c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:29.690 7fde14146c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.691 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:40:29.691 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:40:29.691 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:40:29.692 7fde14146f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:40:29.692 7fde1411c220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.693 7fde1411c220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:29.693 7fde1411c220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.693 7fde1411c220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.693 7fde1411c220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:29.693 7fde1411c220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.697 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:40:29.699 7fde14146f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:40:29.699 7fde1411c9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.700 7fde1411c9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:29.700 7fde1411c9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.735 7fde1420f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:40:29.738 7fde14080400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.738 7fde14080400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:40:29.738 7fde14080400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:29.752 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:29.753 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:29.753 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:29.753 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:29.753 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:29.755 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:29.951 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:29.952 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:29.952 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:29.952 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:29.953 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:29.953 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:30.474 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:30.475 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:30.475 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:30.475 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:30.476 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:30.476 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:30.556 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:31.471 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:31.472 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:31.472 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:31.472 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:31.473 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:31.473 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:31.557 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:32.559 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:33.552 7fde14080400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:40:33.554 7fde14080400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:40:33.554 7fde14080400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:40:33.554 7fde14080400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:40:33.554 7fde14080400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:40:33.554 7fde14080400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:40:33.554 7fde14080400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:40:33.555 7fde14080400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:40:33.555 7fde14080400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:40:33.555 7fde14080400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:40:33.555 7fde14080400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:40:33.555 7fde14080900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:40:33.557 7fde14080900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:40:33.560 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:33.817 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:33.818 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:33.818 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:33.819 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:33.819 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:33.820 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:34.561 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:35.563 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:36.564 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:37.565 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:37.911 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:37.912 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:37.912 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:37.913 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:37.913 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:37.913 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:38.566 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:39.567 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:40.568 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:41.569 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:42.571 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:42.923 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:42.924 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:42.924 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:42.925 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:42.925 7fde141bfc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:42.925 7fde141bfc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:43.572 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:44.573 7fde1420ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:44.633 7fde141bfc20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:40:44.633 7fde141bfc20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:40:44.634 7fde141bfc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:40:44.758 7fde1420f7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:40:44.759 7fde18ca3c40 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:40:44.759 7fde18ca3c40 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:40:44.759 7fde141bf680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:40:44.759 7fde141bf680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:40:44.773 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:40:44.773 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:40:44.773 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:40:44.774 7fde18ca3c40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:40:54.257 7f9de52a6d80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:40:54.257 7f9de52a6d80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:40:54.257 7f9de52a6d80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:40:54.260 7f9de52a6d80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:40:54.261 7f9de08202c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:40:54.261 7f9de08202c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:40:54.261 7f9de08207c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:40:54.261 7f9de08207c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:40:54.262 7f9de08207c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:40:54.262 7f9de08207c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:40:54.262 7f9de08207c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.262 7f9de08207c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:40:54.262 7f9de08207c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.262 7f9de08207c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.262 7f9de08207c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353654262166 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:40:54.262 7f9de08207c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.263 7f9de08207c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.263 7f9de08207c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:54.263 7f9de08207c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.263 7f9de08207c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.263 7f9de08207c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:54.264 7f9de08207c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.264 7f9de08207c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.264 7f9de08207c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:40:54.264 7f9de08207c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.264 7f9de08207c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.264 7f9de08207c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:54.264 7f9de08207c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.264 7f9de0820e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:40:54.264 7f9de0820e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:40:54.265 7f9de0820c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:40:54.266 7f9de07d0540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.266 7f9de07d0540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:40:54.266 7f9de07d0540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.266 7f9de07d0540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.266 7f9de07d0540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:54.266 7f9de07d0540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.321 7f9de08207c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:40:54.328 7f9de08207c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:40:54.329 7f9de2930040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.329 7f9de2930040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:54.329 7f9de2930040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.329 7f9de2930040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.329 7f9de2930040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:54.329 7f9de2930040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.330 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:54.331 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.331 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:54.334 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.334 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:54.335 7f9de07d07c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.335 7f9de07d07c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:54.335 7f9de07d07c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.338 7f9de08207c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:40:54.339 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.339 7f9de07d07c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.339 7f9de07d07c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:54.339 7f9de07d07c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.347 7f9de2930c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.348 7f9de2930c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:54.348 7f9de2930c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.350 7f9de2930c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.350 7f9de2930c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:54.350 7f9de2930c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.385 7f9de08207c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:40:54.386 7f9de2930ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.386 7f9de2930ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:40:54.386 7f9de2930ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.389 7f9de08207c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:40:54.389 7f9de2930ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.390 7f9de2930ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:40:54.390 7f9de2930ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.390 7f9de08207c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:40:54.390 7f9de08207c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:40:54.391 7f9de08207c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:40:54.391 7f9de2930e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:40:54.391 7f9de1df80e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.392 7f9de1df80e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:54.392 7f9de1df80e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.392 7f9de1df80e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.392 7f9de1df80e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:54.392 7f9de1df80e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.395 7f9de08207c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:40:54.399 7f9de2930e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:40:54.399 7f9de1df8860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.399 7f9de1df8860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:40:54.399 7f9de1df8860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.420 7f9de08207c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:40:54.421 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.421 7f9de1dd3400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:40:54.421 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:40:54.477 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:54.478 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.478 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:54.478 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.478 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:54.479 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.668 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:54.669 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.669 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:54.670 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.670 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:54.670 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.980 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:54.981 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.981 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:54.982 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:54.982 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:54.982 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:55.265 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:55.391 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:55.392 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:55.392 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:55.392 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:55.392 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:55.393 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:56.267 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:56.501 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:56.502 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:56.503 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:56.503 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:56.503 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:56.504 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:57.269 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:58.262 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:40:58.263 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:40:58.264 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:40:58.264 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:40:58.264 7f9de1dd3400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:40:58.264 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:40:58.264 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:40:58.264 7f9de1dd3400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:40:58.264 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:40:58.265 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:40:58.265 7f9de1dd3400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:40:58.265 7f9de1dd3900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:40:58.266 7f9de1dd3900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:40:58.271 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:40:58.521 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:40:58.523 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:58.524 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:40:58.524 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:58.525 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:40:58.526 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:40:59.272 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:00.273 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:01.275 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:02.275 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:02.276 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:02.277 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:02.277 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:02.277 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:02.277 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:02.278 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:03.276 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:04.278 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:05.279 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:06.280 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:07.281 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:08.195 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:08.196 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:08.196 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:08.197 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:08.197 7f9de07d0a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:08.198 7f9de07d0a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:08.282 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:09.283 7f9de0820f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:09.303 7f9de07d0a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:41:09.303 7f9de07d0a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:41:09.304 7f9de07d0a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:41:09.443 7f9de08207c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:41:09.443 7f9de52a6d80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:41:09.443 7f9de52a6d80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:41:09.443 7f9de07d0680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:41:09.444 7f9de07d0680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:41:09.464 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:41:09.464 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:41:09.465 7f9de52a6d80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:41:19.092 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:41:19.092 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:41:19.092 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:41:19.096 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:41:19.096 7f50a1a482c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:41:19.096 7f50a1a482c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:41:19.097 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:41:19.097 7f50a1a487c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:41:19.097 7f50a1a487c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:41:19.098 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:41:19.098 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.098 7f50a1a487c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:41:19.098 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.098 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.098 7f50a1a487c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353679097658 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:41:19.098 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.099 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.099 7f50a1a487c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:41:19.099 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.099 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.099 7f50a1a487c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:41:19.100 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.100 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.100 7f50a1a487c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:41:19.100 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.100 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.100 7f50a1a487c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:19.100 7f50a1a487c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.100 7f50a1a48e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:41:19.100 7f50a1a48e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:41:19.101 7f50a1a48c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:41:19.102 7f50a19f8540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.102 7f50a19f8540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:41:19.102 7f50a19f8540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.102 7f50a19f8540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.102 7f50a19f8540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:19.102 7f50a19f8540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.216 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:41:19.237 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:41:19.238 7f50a1994040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.239 7f50a1994040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:41:19.239 7f50a1994040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.239 7f50a1994040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.239 7f50a1994040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:41:19.239 7f50a1994040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.240 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:19.241 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:19.241 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:19.242 7f50a19f87c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.242 7f50a19f87c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:41:19.243 7f50a19f87c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.243 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:19.243 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:19.255 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:19.257 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:41:19.257 7f50a19f87c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.257 7f50a19f87c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:19.257 7f50a19f87c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.259 7f50a1994040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.259 7f50a1994040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:19.259 7f50a1994040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.280 7f50a1994c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.280 7f50a1994c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:19.280 7f50a1994c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.332 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:41:19.332 7f50a1994ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.333 7f50a1994ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:41:19.333 7f50a1994ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.336 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:41:19.336 7f50a1994ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.336 7f50a1994ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:19.336 7f50a1994ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.337 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:41:19.337 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:41:19.338 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:41:19.338 7f50a1994e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:41:19.338 7f50a18c60e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.339 7f50a18c60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:41:19.339 7f50a18c60e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.339 7f50a18c60e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.339 7f50a18c60e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:41:19.339 7f50a18c60e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.343 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:41:19.351 7f50a1994e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:41:19.352 7f50a18c6860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.352 7f50a18c6860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:41:19.353 7f50a18c6860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.393 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:19.394 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:19.394 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:19.394 7f50a1a487c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:41:19.395 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:19.395 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:19.396 7f50a30db4a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.397 7f50a30db4a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:41:19.397 7f50a30db4a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:19.397 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:19.745 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:19.746 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:19.746 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:19.746 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:19.746 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:19.747 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:20.101 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:20.658 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:20.658 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:20.658 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:20.659 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:20.659 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:20.660 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:21.102 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:22.103 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:22.894 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:22.895 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:22.895 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:22.896 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:22.896 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:22.897 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:23.098 7f50a30db540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:41:23.099 7f50a30db540 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:41:23.099 7f50a30db540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:41:23.099 7f50a30db540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:41:23.100 7f50a30db540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:41:23.100 7f50a30db540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:41:23.100 7f50a30db540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:41:23.100 7f50a30db540 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:41:23.100 7f50a30db540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:41:23.100 7f50a30db540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:41:23.100 7f50a30db540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:41:23.101 7f50a30dba40 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:41:23.102 7f50a30dba40 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:41:23.104 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:24.105 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:25.107 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:26.108 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:27.109 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:28.111 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:28.340 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:28.340 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:28.341 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:28.341 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:28.341 7f50a19f8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:28.342 7f50a19f8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:29.113 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:30.114 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:31.116 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:32.117 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:33.118 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:34.119 7f50a1a48f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:34.168 7f50a19f8a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:41:34.169 7f50a19f8a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:41:34.169 7f50a19f8a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:41:34.442 7f50a1a487c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:41:34.443 7f50a64c7ac0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:41:34.443 7f50a64c7ac0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:41:34.443 7f50a21e9c20 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:41:34.443 7f50a21e9c20 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:41:34.459 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:41:34.460 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:41:34.461 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:41:34.461 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:41:34.461 7f50a64c7ac0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:41:43.972 7f397a16fa40 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:41:43.972 7f397a16fa40 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:41:43.972 7f397a16fa40 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:41:43.977 7f397a16fa40 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:41:43.977 7f39751242c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:41:43.977 7f39751242c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:41:43.977 7f39751247c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:41:43.978 7f39751247c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:41:43.978 7f39751247c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:41:43.979 7f39751247c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:41:43.979 7f39751247c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.979 7f39751247c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:41:43.979 7f39751247c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.979 7f39751247c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.979 7f39751247c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353703978790 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:41:43.979 7f39751247c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.980 7f39751247c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.980 7f39751247c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:43.981 7f39751247c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.982 7f3975124e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:41:43.982 7f3975124e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:41:43.983 7f3975124c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:41:43.983 7f39750d4540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.983 7f39750d4540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:41:43.983 7f39750d4540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.983 7f39750d4540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:43.983 7f39750d4540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:43.984 7f39750d4540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.058 7f39751247c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:41:44.068 7f39751247c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:41:44.069 7f3974f9b040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.069 7f3974f9b040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:41:44.069 7f3974f9b040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.069 7f3974f9b040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.069 7f3974f9b040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:41:44.069 7f3974f9b040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.070 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:44.071 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.072 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:44.073 7f39750d47c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.073 7f39750d47c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:41:44.073 7f39750d47c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.073 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.073 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:44.075 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.076 7f39751247c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:41:44.076 7f39750d47c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.076 7f39750d47c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:44.076 7f39750d47c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.081 7f3974f9b040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.081 7f3974f9b040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:44.082 7f3974f9b040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.099 7f3974f9bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.100 7f3974f9bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:44.100 7f3974f9bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.124 7f39751247c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:41:44.125 7f3974f9bae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.125 7f3974f9bae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:41:44.125 7f3974f9bae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.139 7f39751247c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:41:44.139 7f3974f9bae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.139 7f3974f9bae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:41:44.139 7f3974f9bae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.140 7f39751247c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:41:44.140 7f39751247c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:41:44.140 7f39751247c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:41:44.140 7f3974f9be00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:41:44.141 7f39772350e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.141 7f39772350e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:41:44.141 7f39772350e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.141 7f39772350e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.141 7f39772350e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:41:44.141 7f39772350e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.146 7f39751247c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:41:44.148 7f3974f9be00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:41:44.148 7f39772359a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.148 7f39772359a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:41:44.148 7f39772359a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.176 7f39751247c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:41:44.178 7f39772357c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.178 7f39772357c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:41:44.178 7f39772357c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:41:44.204 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:44.205 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.205 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:44.206 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.206 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:44.206 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.409 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:44.410 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.411 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:44.411 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.411 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:44.412 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.828 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:44.829 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.829 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:44.829 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.829 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:44.830 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:44.985 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:45.967 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:45.968 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:45.968 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:45.969 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:45.969 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:45.969 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:45.987 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:46.988 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:47.979 7f39772357c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:41:47.980 7f39772357c0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:41:47.980 7f39772357c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:41:47.980 7f39772357c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:41:47.980 7f39772357c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:41:47.980 7f39772357c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:41:47.980 7f39772357c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:41:47.980 7f39772357c0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:41:47.981 7f39772357c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:41:47.981 7f39772357c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:41:47.981 7f39772357c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:41:47.981 7f3974a39180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:41:47.982 7f3974a39180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:41:47.989 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:48.902 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:48.903 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:48.903 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:48.904 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:48.904 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:48.905 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:48.990 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:49.993 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:50.994 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:51.995 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:52.997 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:53.997 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:54.586 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:41:54.586 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:54.587 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:41:54.587 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:54.587 7f39750d4a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:41:54.588 7f39750d4a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:41:54.999 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:56.000 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:57.001 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:58.002 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:59.004 7f3975124f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:41:59.008 7f39750d4a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:41:59.009 7f39750d4a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:41:59.009 7f39750d4a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:41:59.232 7f39751247c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:41:59.232 7f397a16fa40 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:41:59.232 7f397a16fa40 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:41:59.233 7f39750d4860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:41:59.233 7f39750d4860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:41:59.247 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:41:59.248 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:41:59.248 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:41:59.248 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:41:59.248 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:41:59.248 7f397a16fa40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:42:08.727 7f89b007cdc0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:42:08.727 7f89b007cdc0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:42:08.727 7f89b007cdc0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:42:08.730 7f89b007cdc0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:42:08.731 7f89ab5f52c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:42:08.731 7f89ab5f52c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:42:08.731 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:42:08.732 7f89ab5f57c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:42:08.732 7f89ab5f57c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:42:08.732 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:42:08.732 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.732 7f89ab5f57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:42:08.732 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.732 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.732 7f89ab5f57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353728732258 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.50:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:42:08.733 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.733 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.733 7f89ab5f57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:08.734 7f89ab5f57c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.735 7f89ab5f5f40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:42:08.735 7f89ab5f5f40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:42:08.736 7f89ab5f5e00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:42:08.736 7f89ab5a47c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.737 7f89ab5a47c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:42:08.737 7f89ab5a47c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.737 7f89ab5a47c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.737 7f89ab5a47c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:08.737 7f89ab5a47c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.814 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:42:08.827 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:42:08.829 7f89aaecc0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.829 7f89aaecc0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:08.829 7f89aaecc0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.829 7f89aaecc0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.829 7f89aaecc0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:08.829 7f89aaecc0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.831 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:08.831 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:08.832 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:08.834 7f89ab5a49a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.834 7f89ab5a49a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:08.834 7f89ab5a49a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.834 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:08.834 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:08.835 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:08.836 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:42:08.836 7f89ab5a49a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.836 7f89ab5a49a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:08.837 7f89ab5a49a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.848 7f89aaeccd60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.848 7f89aaeccd60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:08.849 7f89aaeccd60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.850 7f89aaeccd60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.850 7f89aaeccd60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:08.850 7f89aaeccd60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.871 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:42:08.872 7f89aaeccc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.872 7f89aaeccc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:08.872 7f89aaeccc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.876 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:42:08.876 7f89aaeccc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.876 7f89aaeccc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:08.877 7f89aaeccc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.877 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:42:08.877 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:42:08.878 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:42:08.879 7f89aaeccf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:42:08.880 7f89ab472220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.880 7f89ab472220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:08.880 7f89ab472220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.880 7f89ab472220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.881 7f89ab472220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:08.881 7f89ab472220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.885 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:42:08.888 7f89aaeccf40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:42:08.888 7f89ab4729a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.888 7f89ab4729a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:08.888 7f89ab4729a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.912 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:08.913 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:08.913 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:08.914 7f89ab5f57c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:42:08.915 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:08.915 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:08.917 7f89ab580540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.917 7f89ab580540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:42:08.918 7f89ab580540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:08.918 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.055 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:09.056 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.056 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:09.057 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.057 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:09.057 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.312 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:09.313 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.313 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:09.313 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.314 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:09.314 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.736 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:09.981 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:09.981 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.981 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:09.982 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:09.982 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:09.982 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:10.737 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:11.088 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:11.089 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:11.089 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:11.089 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:11.089 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:11.090 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:11.738 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:12.732 7f89ab580540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:42:12.734 7f89ab580540 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:42:12.734 7f89ab580540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:42:12.734 7f89ab580540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:42:12.734 7f89ab580540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:42:12.734 7f89ab580540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:42:12.734 7f89ab580540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:42:12.735 7f89ab580540 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:42:12.735 7f89ab580540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:42:12.735 7f89ab580540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:42:12.735 7f89ab580540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:42:12.735 7f89ab580a40 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:42:12.736 7f89ab580a40 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:42:12.739 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:13.037 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:13.037 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:13.038 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:13.038 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:13.038 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:13.039 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:13.741 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:14.741 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:15.749 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:16.356 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:16.357 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:16.357 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:16.358 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:16.358 7f89ab5a4c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:16.358 7f89ab5a4c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:16.750 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:17.751 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:18.753 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:19.754 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:20.756 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:21.758 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:22.759 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:23.760 7f89ab5a42c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:23.791 7f89ab5a4c20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:42:23.791 7f89ab5a4c20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:42:23.792 7f89ab5a4c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:42:23.930 7f89ab5f57c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:42:23.930 7f89b007cdc0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:42:23.930 7f89b007cdc0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:42:23.930 7f89ab5a4cc0 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:42:23.931 7f89ab5a4cc0 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:42:23.944 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:42:23.944 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:42:23.944 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:42:23.944 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:42:23.944 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:42:23.945 7f89b007cdc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:42:33.473 7f277f481dc0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:42:33.473 7f277f481dc0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:42:33.473 7f277f481dc0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:42:33.482 7f277f481dc0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:42:33.484 7f277aa012c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:42:33.484 7f277aa012c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:42:33.485 7f277aa017c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:42:33.487 7f277aa017c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:42:33.487 7f277aa017c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:42:33.487 7f277aa017c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:42:33.487 7f277aa017c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.488 7f277aa017c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:42:33.488 7f277aa017c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.488 7f277aa017c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.488 7f277aa017c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353753487506 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:42:33.488 7f277aa017c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.491 7f277aa017c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.492 7f277aa017c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.493 7f277aa017c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:33.493 7f277aa017c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.493 7f277aa01e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:42:33.494 7f277aa01e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:42:33.496 7f277aa01c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:42:33.497 7f277a9b1540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.498 7f277a9b1540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:42:33.498 7f277a9b1540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.498 7f277a9b1540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.498 7f277a9b1540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:33.499 7f277a9b1540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.612 7f277aa017c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:42:33.630 7f277aa017c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:42:33.641 7f277a94d040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.641 7f277a94d040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:33.641 7f277a94d040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.641 7f277a94d040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.642 7f277a94d040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:33.642 7f277a94d040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.643 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:33.643 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:33.643 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:33.644 7f277a9b17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.645 7f277a9b17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:33.645 7f277a9b17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.645 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:33.645 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:33.646 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:33.646 7f277aa017c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:42:33.647 7f277a9b17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.647 7f277a9b17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:33.647 7f277a9b17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.666 7f277a94dc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.666 7f277a94dc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:33.666 7f277a94dc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.667 7f277a94dc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.668 7f277a94dc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:33.668 7f277a94dc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.708 7f277aa017c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:42:33.709 7f277a94dae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.709 7f277a94dae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:33.709 7f277a94dae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.713 7f277aa017c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:42:33.713 7f277a94dae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.713 7f277a94dae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:33.713 7f277a94dae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.714 7f277aa017c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:42:33.714 7f277aa017c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:42:33.714 7f277aa017c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:42:33.714 7f277a94de00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:42:33.715 7f277a8850e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.715 7f277a8850e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:33.715 7f277a8850e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.715 7f277a8850e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.716 7f277a8850e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:33.716 7f277a8850e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.719 7f277aa017c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:42:33.722 7f277a94de00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:42:33.722 7f277a885860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.723 7f277a885860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:33.723 7f277a885860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.769 7f277aa017c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:42:33.772 7f277c095400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.772 7f277c095400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:42:33.772 7f277c095400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:33.778 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:33.779 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:33.779 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:33.780 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:33.780 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:33.782 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.006 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:34.007 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.007 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:34.008 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.008 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:34.008 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.288 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:34.288 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.289 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:34.289 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.289 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:34.290 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.495 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:34.677 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:34.678 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.679 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:34.680 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:34.680 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:34.680 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:35.496 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:35.761 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:35.762 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:35.762 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:35.763 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:35.763 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:35.763 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:36.498 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:37.487 7f277c095400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:42:37.489 7f277c095400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:42:37.489 7f277c095400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:42:37.489 7f277c095400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:42:37.489 7f277c095400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:42:37.489 7f277c095400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:42:37.489 7f277c095400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:42:37.490 7f277c095400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:42:37.490 7f277c095400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:42:37.490 7f277c095400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:42:37.490 7f277c095400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:42:37.490 7f277c095900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:42:37.491 7f277c095900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:42:37.498 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:37.577 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:37.578 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:37.578 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:37.579 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:37.579 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:37.579 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:38.499 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:39.500 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:40.501 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:41.016 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:41.017 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:41.018 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:41.018 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:41.018 7f277a9b1a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:41.019 7f277a9b1a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:41.503 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:42.504 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:43.506 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:44.507 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:45.508 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:46.509 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:47.510 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:48.511 7f277aa01f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:42:48.553 7f277a9b1a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:42:48.553 7f277a9b1a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:42:48.554 7f277a9b1a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:42:48.863 7f277aa017c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:42:48.863 7f277f481dc0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:42:48.864 7f277f481dc0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:42:48.864 7f277a9b1680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:42:48.864 7f277a9b1680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:42:48.880 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:42:48.880 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:42:48.880 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:42:48.881 7f277f481dc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:42:58.811 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:42:58.811 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:42:58.811 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:42:58.815 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:42:58.815 7f94c9f4f2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:42:58.816 7f94c9f4f2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:42:58.816 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:42:58.816 7f94c9f4f7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:42:58.817 7f94c9f4f7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:42:58.817 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:42:58.817 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.817 7f94c9f4f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:42:58.817 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.817 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.817 7f94c9f4f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353778817181 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:42:58.817 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.819 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.819 7f94c9f4f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:58.819 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.819 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.819 7f94c9f4f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:58.819 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.819 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.819 7f94c9f4f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:42:58.820 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.820 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.820 7f94c9f4f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:58.820 7f94c9f4f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.820 7f94c9f4fe00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:42:58.821 7f94c9f4fe00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:42:58.822 7f94c9f4fc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:42:58.822 7f94c9eff540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.823 7f94c9eff540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:42:58.823 7f94c9eff540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.823 7f94c9eff540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.823 7f94c9eff540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:58.823 7f94c9eff540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.887 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:42:58.898 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:42:58.899 7f94c9ede040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.899 7f94c9ede040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:58.899 7f94c9ede040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.899 7f94c9ede040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.899 7f94c9ede040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:58.899 7f94c9ede040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.900 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:58.901 7f94c9eff7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.902 7f94c9eff7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:58.902 7f94c9eff7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.902 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:58.902 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:58.903 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:58.903 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:58.904 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:42:58.904 7f94c9eff7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.904 7f94c9eff7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:58.905 7f94c9eff7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.905 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:58.913 7f94c9edec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.913 7f94c9edec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:58.913 7f94c9edec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.914 7f94c9edec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.915 7f94c9edec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:58.915 7f94c9edec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.955 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:42:58.956 7f94c9edeae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.956 7f94c9edeae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:42:58.957 7f94c9edeae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.962 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:42:58.962 7f94c9edeae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.963 7f94c9edeae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:42:58.963 7f94c9edeae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.963 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:42:58.963 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:42:58.964 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:42:58.964 7f94c9edee00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:42:58.965 7f94c981e0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.965 7f94c981e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:58.965 7f94c981e0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.965 7f94c981e0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.965 7f94c981e0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:58.965 7f94c981e0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.970 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:42:58.972 7f94c9edee00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:42:58.973 7f94c981e9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.973 7f94c981e9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:42:58.973 7f94c981e9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:58.999 7f94c9f4f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:42:59.001 7f94cc059400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:59.002 7f94cc059400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:42:59.002 7f94cc059400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:42:59.016 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:59.017 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.017 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:59.017 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.017 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:59.018 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.187 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:59.188 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.188 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:59.189 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.189 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:59.189 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.598 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:42:59.599 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.599 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:42:59.600 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.600 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:42:59.601 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:42:59.822 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:00.455 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:00.456 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:00.456 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:00.456 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:00.457 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:00.457 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:00.823 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:01.824 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:02.650 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:02.651 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:02.651 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:02.651 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:02.652 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:02.652 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:02.817 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:43:02.819 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:43:02.819 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:43:02.819 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:43:02.819 7f94c981e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:43:02.819 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:43:02.819 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:43:02.819 7f94c981e7c0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:43:02.820 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:43:02.820 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:43:02.820 7f94c981e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:43:02.820 7f94c981ec20 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:43:02.821 7f94c981ec20 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:43:02.825 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:03.826 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:04.828 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:05.829 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:05.923 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:05.923 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:05.924 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:05.924 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:05.924 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:05.925 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:06.830 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:07.831 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:08.832 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:09.833 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:10.343 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:10.344 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:10.344 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:10.345 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:10.345 7f94c9effa40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:10.346 7f94c9effa40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:10.835 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:11.836 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:12.838 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:13.839 7f94c9f4ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:13.861 7f94c9effa40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:43:13.861 7f94c9effa40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:43:13.862 7f94c9effa40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:43:14.046 7f94c9f4f7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:43:14.046 7f94ce9e0cc0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:43:14.046 7f94ce9e0cc0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:43:14.046 7f94c9eff860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:43:14.047 7f94c9eff860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:43:14.064 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:43:14.064 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:43:14.064 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:43:14.064 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:43:14.064 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:43:14.064 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:43:14.064 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:43:14.065 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:43:14.065 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:43:14.065 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:43:14.065 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:43:14.065 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:43:14.065 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:43:14.065 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:43:14.065 7f94ce9e0cc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:43:23.129 7f22f9cfc700 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:43:23.129 7f22f9cfc700 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:43:23.129 7f22f9cfc700 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:43:23.132 7f22f9cfc700 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:43:23.133 7f22f527d2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:43:23.133 7f22f527d2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:43:23.133 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:43:23.133 7f22f527d7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:43:23.134 7f22f527d7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:43:23.134 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:43:23.134 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.134 7f22f527d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:43:23.134 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.134 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.134 7f22f527d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353803134087 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:43:23.134 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.137 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.137 7f22f527d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:43:23.137 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.137 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.137 7f22f527d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:43:23.137 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.137 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.138 7f22f527d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:43:23.138 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.138 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.138 7f22f527d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:23.138 7f22f527d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.140 7f22f527df40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:43:23.140 7f22f527df40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:43:23.142 7f22f527de00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:43:23.143 7f22f522d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.143 7f22f522d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:43:23.143 7f22f522d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.143 7f22f522d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.143 7f22f522d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:23.143 7f22f522d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.216 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:43:23.225 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:43:23.226 7f22f51c70e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.227 7f22f51c70e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:43:23.227 7f22f51c70e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.227 7f22f51c70e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.227 7f22f51c70e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:43:23.227 7f22f51c70e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.228 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:23.229 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.229 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:23.231 7f22f522d9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.231 7f22f522d9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:43:23.231 7f22f522d9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.231 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.231 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:23.232 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:43:23.233 7f22f522d9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.233 7f22f522d9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:23.233 7f22f522d9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.233 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.242 7f22f51c7d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.243 7f22f51c7d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:23.243 7f22f51c7d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.244 7f22f51c7d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.244 7f22f51c7d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:23.244 7f22f51c7d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.314 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:23.315 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.316 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:23.316 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.316 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:23.317 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.330 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:43:23.330 7f22f51c7c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.331 7f22f51c7c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:43:23.331 7f22f51c7c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.334 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:43:23.334 7f22f51c7c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.334 7f22f51c7c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:23.335 7f22f51c7c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.335 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:43:23.335 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:43:23.335 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:43:23.336 7f22f51c7f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:43:23.336 7f22f5100220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.337 7f22f5100220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:43:23.337 7f22f5100220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.337 7f22f5100220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.338 7f22f5100220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:43:23.338 7f22f5100220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.343 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:43:23.353 7f22f51c7f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:43:23.353 7f22f51009a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.354 7f22f51009a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:43:23.354 7f22f51009a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.404 7f22f527d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:43:23.406 7f22f6911400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.406 7f22f6911400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:43:23.406 7f22f6911400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:23.506 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:23.506 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.506 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:23.507 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.507 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:23.507 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.780 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:23.781 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.781 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:23.781 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:23.782 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:23.782 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:24.141 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:24.468 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:24.469 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:24.469 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:24.469 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:24.470 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:24.470 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:25.142 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:25.683 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:25.684 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:25.684 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:25.685 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:25.685 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:25.685 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:26.144 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:27.133 7f22f6911400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:43:27.135 7f22f6911400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:43:27.135 7f22f6911400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:43:27.135 7f22f6911400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:43:27.135 7f22f6911400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:43:27.135 7f22f6911400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:43:27.135 7f22f6911400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:43:27.136 7f22f6911400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:43:27.136 7f22f6911400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:43:27.136 7f22f6911400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:43:27.136 7f22f6911400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:43:27.136 7f22f6911900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:43:27.137 7f22f6911900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:43:27.145 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:28.146 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:28.895 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:28.896 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:28.896 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:28.897 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:28.897 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:28.897 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:29.146 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:30.148 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:31.148 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:32.149 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:33.151 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:34.152 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:35.154 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:36.154 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:37.155 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:37.707 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:37.707 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:37.707 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:37.708 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:37.708 7f22f522dc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:37.709 7f22f522dc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:38.157 7f22f522d2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:38.211 7f22f522dc20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:43:38.212 7f22f522dc20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:43:38.212 7f22f522dc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:43:38.459 7f22f527d7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:43:38.459 7f22f9cfc700 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:43:38.459 7f22f9cfc700 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:43:38.460 7f22f522d860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:43:38.460 7f22f522d860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:43:38.474 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:43:38.474 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:43:38.474 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:43:38.474 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:43:38.474 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:43:38.474 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:43:38.474 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:43:38.475 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:43:38.475 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:43:38.475 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:43:38.475 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:43:38.475 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:43:38.475 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:43:38.475 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:43:38.475 7f22f9cfc700 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:43:47.812 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:43:47.812 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:43:47.812 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:43:47.816 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:43:47.817 7fcd251912c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:43:47.817 7fcd251912c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:43:47.817 7fcd251917c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:43:47.817 7fcd251917c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:43:47.818 7fcd251917c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:43:47.818 7fcd251917c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:43:47.818 7fcd251917c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.818 7fcd251917c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:43:47.818 7fcd251917c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.818 7fcd251917c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.818 7fcd251917c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353827818114 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:43:47.818 7fcd251917c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.819 7fcd251917c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.819 7fcd251917c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:43:47.819 7fcd251917c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.819 7fcd251917c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.820 7fcd251917c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:43:47.820 7fcd251917c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.820 7fcd251917c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.820 7fcd251917c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:43:47.820 7fcd251917c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.820 7fcd251917c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.820 7fcd251917c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:47.820 7fcd251917c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.820 7fcd25191e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:43:47.821 7fcd25191e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:43:47.822 7fcd25191c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:43:47.822 7fcd25141540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.822 7fcd25141540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:43:47.822 7fcd25141540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.822 7fcd25141540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.823 7fcd25141540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:47.823 7fcd25141540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.898 7fcd251917c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:43:47.908 7fcd251917c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:43:47.909 7fcd25097040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.909 7fcd25097040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:43:47.909 7fcd25097040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.909 7fcd25097040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.909 7fcd25097040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:43:47.910 7fcd25097040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.913 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:47.914 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:47.914 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:47.917 7fcd251417c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.917 7fcd251417c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:43:47.917 7fcd251417c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.918 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:47.918 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:47.919 7fcd251917c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:43:47.920 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:47.920 7fcd251417c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.920 7fcd251417c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:47.920 7fcd251417c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.929 7fcd25097c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.931 7fcd25097c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:47.931 7fcd25097c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.932 7fcd25097c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.932 7fcd25097c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:47.932 7fcd25097c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.967 7fcd251917c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:43:47.968 7fcd25097ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.968 7fcd25097ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:43:47.968 7fcd25097ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.976 7fcd251917c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:43:47.977 7fcd25097ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.977 7fcd25097ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:43:47.977 7fcd25097ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.978 7fcd251917c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:43:47.978 7fcd251917c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:43:47.978 7fcd251917c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:43:47.979 7fcd25097e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:43:47.979 7fcd251030e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.980 7fcd251030e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:43:47.980 7fcd251030e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.980 7fcd251030e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.980 7fcd251030e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:43:47.980 7fcd251030e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.984 7fcd251917c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:43:47.986 7fcd25097e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:43:47.986 7fcd25103860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.986 7fcd25103860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:43:47.986 7fcd25103860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:47.989 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:47.990 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:47.990 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:47.991 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:47.991 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:47.992 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:48.011 7fcd251917c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:43:48.013 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:48.013 7fcd24a6c400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:43:48.013 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:43:48.143 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:48.144 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:48.144 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:48.145 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:48.145 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:48.145 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:48.503 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:48.504 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:48.504 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:48.504 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:48.504 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:48.505 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:48.821 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:49.384 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:49.384 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:49.384 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:49.385 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:49.385 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:49.386 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:49.823 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:50.824 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:51.395 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:51.396 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:51.396 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:51.397 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:51.397 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:51.397 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:51.818 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:43:51.819 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:43:51.820 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:43:51.820 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:43:51.820 7fcd24a6c400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:43:51.820 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:43:51.820 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:43:51.820 7fcd24a6c400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:43:51.821 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:43:51.821 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:43:51.821 7fcd24a6c400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:43:51.821 7fcd24a6c900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:43:51.823 7fcd24a6c900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:43:51.826 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:52.827 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:53.828 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:54.830 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:55.831 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:56.059 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:43:56.060 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:56.060 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:43:56.060 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:56.060 7fcd25141a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:43:56.061 7fcd25141a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:43:56.833 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:57.834 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:58.835 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:43:59.836 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:00.839 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:01.840 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:02.841 7fcd25191f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:02.890 7fcd25141a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:44:02.891 7fcd25141a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:44:02.891 7fcd25141a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:44:03.077 7fcd251917c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:44:03.077 7fcd2a1e5940 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:44:03.077 7fcd2a1e5940 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:44:03.077 7fcd25141680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:44:03.077 7fcd25141680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:44:03.092 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:44:03.092 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:44:03.093 7fcd2a1e5940 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:44:12.420 7f54362a2f80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:44:12.420 7f54362a2f80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:44:12.420 7f54362a2f80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:44:12.424 7f54362a2f80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:44:12.424 7f54316512c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:44:12.424 7f54316512c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:44:12.424 7f54316517c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:44:12.425 7f54316517c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:44:12.425 7f54316517c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:44:12.425 7f54316517c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:44:12.425 7f54316517c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.425 7f54316517c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:44:12.425 7f54316517c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.426 7f54316517c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.426 7f54316517c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353852425445 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:44:12.426 7f54316517c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.426 7f54316517c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:12.427 7f54316517c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.428 7f5431651e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:44:12.428 7f5431651e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:44:12.429 7f5431651c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:44:12.429 7f5431601540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.429 7f5431601540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:44:12.429 7f5431601540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.429 7f5431601540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.429 7f5431601540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:12.429 7f5431601540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.486 7f54316517c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:44:12.494 7f54316517c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:44:12.495 7f5433761040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.495 7f5433761040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:44:12.495 7f5433761040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.495 7f5433761040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.496 7f5433761040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:44:12.496 7f5433761040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.497 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:12.497 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.498 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:12.498 7f54316017c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.499 7f54316017c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:44:12.499 7f54316017c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.500 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.500 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:12.500 7f54316517c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:44:12.501 7f54316017c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.501 7f54316017c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:12.501 7f54316017c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.501 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.508 7f5433761c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.509 7f5433761c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:12.509 7f5433761c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.510 7f5433761c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.510 7f5433761c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:12.510 7f5433761c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.536 7f54316517c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:44:12.537 7f5433761ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.537 7f5433761ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:44:12.538 7f5433761ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.552 7f54316517c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:44:12.553 7f5433761ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.553 7f5433761ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:12.553 7f5433761ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.553 7f54316517c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:44:12.553 7f54316517c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:44:12.554 7f54316517c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:44:12.554 7f5433761e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:44:12.555 7f5432c2b0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.555 7f5432c2b0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:44:12.555 7f5432c2b0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.555 7f5432c2b0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.555 7f5432c2b0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:44:12.555 7f5432c2b0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.560 7f54316517c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:44:12.564 7f5433761e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:44:12.565 7f5432c2bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.565 7f5432c2bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:44:12.565 7f5432c2bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.573 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:12.585 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.586 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:12.586 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.587 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:12.588 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.590 7f54316517c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:44:12.592 7f5432c02400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.592 7f5432c02400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:44:12.593 7f5432c02400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:12.719 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:12.720 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.720 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:12.721 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.721 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:12.721 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.997 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:12.998 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.998 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:12.998 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:12.998 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:12.999 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:13.428 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:13.738 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:13.739 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:13.739 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:13.740 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:13.740 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:13.740 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:14.429 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:15.053 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:15.054 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:15.054 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:15.055 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:15.055 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:15.055 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:15.431 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:16.425 7f5432c2b900 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:44:16.427 7f5432c2b900 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:44:16.427 7f5432c2b900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:44:16.427 7f5432c2b900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:44:16.427 7f5432c2b900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:44:16.427 7f5432c2b900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:44:16.427 7f5432c2b900 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:44:16.428 7f5432c2b900 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:44:16.428 7f5432c2b900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:44:16.428 7f5432c2b900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:44:16.428 7f5432c2b900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:44:16.428 7f5432c2b7c0 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:44:16.430 7f5432c2b7c0 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:44:16.432 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:16.676 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:16.676 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:16.677 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:16.677 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:16.677 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:16.678 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:17.433 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:18.434 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:19.001 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:19.002 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:19.002 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:19.002 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:19.003 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:19.003 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:19.435 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:20.436 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:21.438 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:22.438 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:23.440 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:24.303 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:24.304 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:24.304 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:24.305 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:24.305 7f5431601a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:24.305 7f5431601a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:24.441 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:25.444 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:26.445 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:27.446 7f5431651f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:27.522 7f5431601a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:44:27.522 7f5431601a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:44:27.522 7f5431601a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:44:27.674 7f54316517c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:44:27.675 7f54362a2f80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:44:27.675 7f54362a2f80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:44:27.675 7f5431601860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:44:27.675 7f5431601860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:44:27.732 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:44:27.732 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:44:27.732 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:44:27.733 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:44:27.734 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:44:27.734 7f54362a2f80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:44:36.898 7f416bdebf00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:44:36.898 7f416bdebf00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:44:36.898 7f416bdebf00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:44:36.901 7f416bdebf00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:44:36.901 7f416701f2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:44:36.901 7f416701f2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:44:36.902 7f416701f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:44:36.902 7f416701f7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:44:36.902 7f416701f7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:44:36.902 7f416701f7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:44:36.903 7f416701f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.903 7f416701f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:44:36.903 7f416701f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.903 7f416701f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.903 7f416701f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353876902651 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:44:36.903 7f416701f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:36.904 7f416701f7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.905 7f416701fe00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:44:36.905 7f416701fe00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:44:36.906 7f416701fc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:44:36.906 7f4166fce540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.906 7f4166fce540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:44:36.906 7f4166fce540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.907 7f4166fce540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.907 7f4166fce540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:36.907 7f4166fce540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.961 7f416701f7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:44:36.969 7f416701f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:44:36.970 7f4166fb5040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.970 7f4166fb5040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:44:36.970 7f4166fb5040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.970 7f4166fb5040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.970 7f4166fb5040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:44:36.970 7f4166fb5040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.971 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:36.972 7f4166fce7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.972 7f4166fce7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:44:36.972 7f4166fce7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.973 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:36.973 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:36.974 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:36.974 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:36.975 7f416701f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:44:36.975 7f4166fce7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.975 7f4166fce7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:36.975 7f4166fce7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.976 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:36.977 7f4166fb5040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.977 7f4166fb5040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:36.978 7f4166fb5040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.985 7f4166fb5c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:36.986 7f4166fb5c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:36.986 7f4166fb5c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.037 7f416701f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:44:37.038 7f4166fb5ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.038 7f4166fb5ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:44:37.038 7f4166fb5ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.042 7f416701f7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:44:37.043 7f4166fb5ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.043 7f4166fb5ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:44:37.043 7f4166fb5ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.043 7f416701f7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:44:37.043 7f416701f7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:44:37.044 7f416701f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:44:37.044 7f4166fb5e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:44:37.045 7f4166e990e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.045 7f4166e990e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:44:37.045 7f4166e990e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.045 7f4166e990e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.045 7f4166e990e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:44:37.045 7f4166e990e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.050 7f416701f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:44:37.053 7f4166fb5e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:44:37.054 7f4166e99860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.054 7f4166e99860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:44:37.054 7f4166e99860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.082 7f416701f7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:44:37.085 7f41686b5400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.085 7f41686b5400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:44:37.086 7f41686b5400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:44:37.088 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:37.089 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.089 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:37.091 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.091 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:37.092 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.282 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:37.283 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.283 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:37.283 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.284 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:37.284 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.772 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:37.773 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.773 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:37.773 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.773 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:37.774 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:37.905 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:38.691 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:38.692 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:38.692 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:38.692 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:38.693 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:38.693 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:38.906 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:39.907 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:40.293 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:40.294 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:40.294 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:40.295 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:40.295 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:40.295 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:40.902 7f41686b5400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:44:40.904 7f41686b5400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:44:40.904 7f41686b5400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:44:40.904 7f41686b5400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:44:40.904 7f41686b5400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:44:40.904 7f41686b5400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:44:40.904 7f41686b5400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:44:40.905 7f41686b5400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:44:40.905 7f41686b5400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:44:40.905 7f41686b5400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:44:40.905 7f41686b5400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:44:40.906 7f41686b5900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:44:40.907 7f41686b5900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:44:40.909 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:41.910 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:42.911 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:43.912 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:44.212 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:44:44.212 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:44.213 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:44:44.213 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:44.213 7f4166fcea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:44:44.214 7f4166fcea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:44:44.913 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:45.915 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:46.916 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:47.917 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:48.918 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:49.919 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:50.920 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:51.921 7f416701ff40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:44:51.947 7f4166fcea40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:44:51.947 7f4166fcea40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:44:51.948 7f4166fcea40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:44:52.113 7f416701f7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:44:52.114 7f416bdebf00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:44:52.114 7f416bdebf00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:44:52.114 7f4166fce860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:44:52.114 7f4166fce860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:44:52.134 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:44:52.134 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:44:52.135 7f416bdebf00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:45:01.778 7f1edab3dc80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:45:01.778 7f1edab3dc80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:45:01.778 7f1edab3dc80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:45:01.782 7f1edab3dc80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:45:01.783 7f1ed60ba2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:45:01.783 7f1ed60ba2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:45:01.783 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:45:01.783 7f1ed60ba7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:45:01.784 7f1ed60ba7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:45:01.784 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:45:01.784 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.784 7f1ed60ba7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:45:01.784 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.785 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.785 7f1ed60ba7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353901784320 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:45:01.785 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.785 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.786 7f1ed60ba7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:01.786 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.786 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.786 7f1ed60ba7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:01.786 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.786 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.786 7f1ed60ba7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:01.786 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.787 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.787 7f1ed60ba7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:01.787 7f1ed60ba7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.787 7f1ed60bae00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:45:01.787 7f1ed60bae00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:45:01.788 7f1ed60bac20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:45:01.789 7f1ed606a540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.789 7f1ed606a540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:45:01.789 7f1ed606a540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.789 7f1ed606a540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.790 7f1ed606a540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:01.790 7f1ed606a540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.873 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:45:01.882 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:45:01.883 7f1ed5fc20e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.883 7f1ed5fc20e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:01.883 7f1ed5fc20e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.883 7f1ed5fc20e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.883 7f1ed5fc20e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:01.884 7f1ed5fc20e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.885 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:01.886 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:01.886 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:01.887 7f1ed606a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.887 7f1ed606a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:01.888 7f1ed606a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.888 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:01.888 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:01.889 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:01.890 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:45:01.890 7f1ed606a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.890 7f1ed606a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:01.890 7f1ed606a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.892 7f1ed606a7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.892 7f1ed606a7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:01.892 7f1ed606a7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.900 7f1ed5fc2d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.900 7f1ed5fc2d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:01.900 7f1ed5fc2d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.936 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:45:01.937 7f1ed5fc2c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.937 7f1ed5fc2c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:01.937 7f1ed5fc2c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.944 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:45:01.944 7f1ed5fc2c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.945 7f1ed5fc2c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:01.945 7f1ed5fc2c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.945 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:45:01.945 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:45:01.946 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:45:01.946 7f1ed5fc2f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:45:01.947 7f1ed5f3f220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.947 7f1ed5f3f220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:01.947 7f1ed5f3f220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.947 7f1ed5f3f220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.947 7f1ed5f3f220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:01.947 7f1ed5f3f220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.952 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:45:01.954 7f1ed5fc2f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:45:01.954 7f1ed5f3f9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.955 7f1ed5f3f9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:01.955 7f1ed5f3f9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.980 7f1ed60ba7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:45:01.982 7f1ed774d400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.982 7f1ed774d400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:45:01.982 7f1ed774d400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:01.999 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:02.000 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:02.000 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:02.001 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:02.001 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:02.001 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:02.273 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:02.274 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:02.274 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:02.274 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:02.275 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:02.275 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:02.789 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:02.825 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:02.826 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:02.826 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:02.827 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:02.827 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:02.827 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:03.790 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:04.364 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:04.365 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:04.365 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:04.366 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:04.366 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:04.366 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:04.791 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:05.784 7f1ed774d400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:45:05.786 7f1ed774d400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:45:05.786 7f1ed774d400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:45:05.786 7f1ed774d400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:45:05.787 7f1ed774d400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:45:05.787 7f1ed774d400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:45:05.787 7f1ed774d400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:45:05.787 7f1ed774d400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:45:05.787 7f1ed774d400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:45:05.788 7f1ed774d400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:45:05.788 7f1ed774d400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:45:05.788 7f1ed774d900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:45:05.789 7f1ed774d900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:45:05.793 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:06.794 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:07.795 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:08.056 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:08.057 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:08.057 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:08.058 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:08.058 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:08.059 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:08.797 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:09.797 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:10.799 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:11.800 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:12.692 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:12.693 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:12.693 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:12.694 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:12.694 7f1ed606ac20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:12.695 7f1ed606ac20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:12.801 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:13.802 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:14.803 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:15.805 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:16.807 7f1ed60baf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:16.815 7f1ed606ac20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:45:16.815 7f1ed606ac20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:45:16.816 7f1ed606ac20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:45:17.018 7f1ed60ba7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:45:17.019 7f1edab3dc80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:45:17.019 7f1edab3dc80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:45:17.019 7f1ed606a680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:45:17.019 7f1ed606a680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:45:17.033 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:45:17.033 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:45:17.033 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:45:17.033 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:45:17.034 7f1edab3dc80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:45:27.233 7fbaab507a80 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:45:27.233 7fbaab507a80 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:45:27.233 7fbaab507a80 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:45:27.237 7fbaab507a80 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:45:27.237 7fbaa64a72c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:45:27.237 7fbaa64a72c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:45:27.238 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:45:27.238 7fbaa64a77c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:45:27.238 7fbaa64a77c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:45:27.238 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:45:27.238 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.239 7fbaa64a77c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:45:27.239 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.239 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.239 7fbaa64a77c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353927238656 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.189:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:45:27.239 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:27.240 7fbaa64a77c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.241 7fbaa64a7e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:45:27.241 7fbaa64a7e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:45:27.242 7fbaa64a7c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:45:27.243 7fbaa6456540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.243 7fbaa6456540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:45:27.243 7fbaa6456540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.243 7fbaa6456540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.243 7fbaa6456540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:27.244 7fbaa6456540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.302 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:45:27.309 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:45:27.310 7fbaa63f1040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.310 7fbaa63f1040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:27.311 7fbaa63f1040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.311 7fbaa63f1040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.311 7fbaa63f1040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:27.311 7fbaa63f1040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.313 7fbaa64567c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.314 7fbaa64567c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:27.314 7fbaa64567c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.315 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:27.315 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:45:27.316 7fbaa64567c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.316 7fbaa64567c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:27.316 7fbaa64567c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.317 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.317 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:27.323 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.324 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:27.325 7fbaa63f1d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.325 7fbaa63f1d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:27.325 7fbaa63f1d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.326 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.326 7fbaa63f1d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.326 7fbaa63f1d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:27.326 7fbaa63f1d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.353 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:45:27.354 7fbaa63f1c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.354 7fbaa63f1c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:27.354 7fbaa63f1c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.358 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:45:27.358 7fbaa63f1c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.358 7fbaa63f1c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:27.358 7fbaa63f1c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.359 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:45:27.359 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:45:27.359 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:45:27.359 7fbaa63f1f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:45:27.360 7fbaa632b220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.360 7fbaa632b220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:27.360 7fbaa632b220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.360 7fbaa632b220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.360 7fbaa632b220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:27.360 7fbaa632b220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.364 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:45:27.365 7fbaa63f1f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:45:27.366 7fbaa632b9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.366 7fbaa632b9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:27.366 7fbaa632b9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.388 7fbaa64a77c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:45:27.391 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.391 7fbaa7b3b400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:45:27.391 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:27.423 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:27.424 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.424 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:27.425 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.425 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:27.425 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.629 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:27.630 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.630 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:27.631 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.631 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:27.631 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.968 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:27.969 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.969 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:27.970 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:27.970 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:27.971 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:28.243 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:28.681 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:28.682 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:28.682 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:28.683 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:28.683 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:28.683 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:29.244 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:30.245 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:30.548 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:30.549 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:30.549 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:30.550 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:30.550 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:30.550 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:31.238 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:45:31.240 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:45:31.240 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:45:31.240 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:45:31.240 7fbaa7b3b400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:45:31.240 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:45:31.240 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:45:31.241 7fbaa7b3b400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:45:31.241 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:45:31.241 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:45:31.241 7fbaa7b3b400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:45:31.241 7fbaa7b3b900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:45:31.243 7fbaa7b3b900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:45:31.247 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:32.248 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:33.074 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:33.074 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:33.075 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:33.078 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:33.079 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:33.079 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:33.249 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:34.250 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:35.251 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:36.253 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:36.856 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:36.857 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:36.857 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:36.858 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:36.858 7fbaa6456a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:36.859 7fbaa6456a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:37.253 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:38.255 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:39.256 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:40.257 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:41.258 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:42.258 7fbaa64a7f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:42.283 7fbaa6456a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:45:42.283 7fbaa6456a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:45:42.284 7fbaa6456a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:45:42.416 7fbaa64a77c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:45:42.416 7fbaab507a80 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:45:42.416 7fbaab507a80 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:45:42.416 7fbaa6456680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:45:42.416 7fbaa6456680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:45:42.430 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:45:42.430 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:45:42.430 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:45:42.430 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:45:42.430 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:45:42.431 7fbaab507a80 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:45:51.852 7f5beef879c0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:45:51.852 7f5beef879c0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:45:51.852 7f5beef879c0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:45:51.856 7f5beef879c0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:45:51.857 7f5be9f382c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:45:51.857 7f5be9f382c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:45:51.857 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:45:51.857 7f5be9f387c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:45:51.858 7f5be9f387c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:45:51.858 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:45:51.858 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.858 7f5be9f387c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:45:51.858 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.858 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.858 7f5be9f387c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353951858191 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.50:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:45:51.858 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.859 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.859 7f5be9f387c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:51.859 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.859 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.859 7f5be9f387c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:51.859 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.859 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.860 7f5be9f387c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:45:51.860 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.860 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.860 7f5be9f387c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:51.860 7f5be9f387c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.860 7f5be9f38e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:45:51.860 7f5be9f38e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:45:51.861 7f5be9f38c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:45:51.862 7f5be9ee8540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.862 7f5be9ee8540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:45:51.862 7f5be9ee8540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.862 7f5be9ee8540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.862 7f5be9ee8540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:51.862 7f5be9ee8540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.919 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:45:51.928 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:45:51.929 7f5bec047040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.929 7f5bec047040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:51.929 7f5bec047040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.929 7f5bec047040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.929 7f5bec047040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:51.929 7f5bec047040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.930 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:51.931 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:51.931 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:51.939 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:51.939 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:51.940 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:51.941 7f5be9ee87c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.941 7f5be9ee87c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:51.941 7f5be9ee87c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.942 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:45:51.942 7f5be9ee87c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.942 7f5be9ee87c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:51.942 7f5be9ee87c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.951 7f5bec047c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.951 7f5bec047c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:51.951 7f5bec047c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.952 7f5bec047c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.952 7f5bec047c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:51.952 7f5bec047c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.980 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:45:51.980 7f5bec047ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.981 7f5bec047ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:45:51.981 7f5bec047ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.984 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:45:51.985 7f5bec047ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.985 7f5bec047ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:45:51.985 7f5bec047ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.985 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:45:51.985 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:45:51.985 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:45:51.986 7f5bec047e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:45:51.986 7f5be9ec00e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.986 7f5be9ec00e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:51.986 7f5be9ec00e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.987 7f5be9ec00e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.987 7f5be9ec00e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:51.987 7f5be9ec00e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.990 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:45:51.994 7f5bec047e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:45:51.995 7f5be9ec0860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:51.996 7f5be9ec0860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:45:51.996 7f5be9ec0860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:52.020 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:52.020 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.021 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:52.021 7f5be9f387c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:45:52.022 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.022 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:52.022 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.024 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:52.024 7f5beb4ea360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:45:52.024 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:45:52.221 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:52.222 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.222 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:52.222 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.222 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:52.223 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.746 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:52.747 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.747 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:52.748 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.748 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:52.748 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:52.861 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:53.460 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:53.461 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:53.461 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:53.462 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:53.462 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:53.462 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:53.863 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:54.864 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:55.421 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:55.422 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:55.422 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:55.422 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:55.422 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:55.423 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:55.858 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:45:55.860 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:45:55.860 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:45:55.860 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:45:55.860 7f5beb4ea360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:45:55.860 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:45:55.860 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:45:55.860 7f5beb4ea360 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:45:55.861 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:45:55.861 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:45:55.861 7f5beb4ea360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:45:55.861 7f5beb4ea900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:45:55.862 7f5beb4ea900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:45:55.864 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:56.865 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:57.866 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:58.716 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:45:58.717 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:58.717 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:45:58.717 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:58.717 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:45:58.718 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:45:58.867 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:45:59.868 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:00.869 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:01.871 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:02.872 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:03.873 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:04.486 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:04.487 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:04.488 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:04.488 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:04.488 7f5be9ee8a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:04.489 7f5be9ee8a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:04.874 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:05.875 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:06.876 7f5be9f38f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:06.900 7f5be9ee8a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:46:06.900 7f5be9ee8a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:46:06.901 7f5be9ee8a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:46:07.033 7f5be9f387c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:46:07.034 7f5beef879c0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:46:07.034 7f5beef879c0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:46:07.034 7f5be9ee8680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:46:07.034 7f5be9ee8680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:46:07.048 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:46:07.049 7f5beef879c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:46:16.653 7f07e1af2b40 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:46:16.653 7f07e1af2b40 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:46:16.655 7f07e1af2b40 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:46:16.659 7f07e1af2b40 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:46:16.660 7f07dca902c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:46:16.660 7f07dca902c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:46:16.660 7f07dca907c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789353976661297 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.50:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:46:16.661 7f07dca907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.664 7f07dca907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.664 7f07dca907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:46:16.664 7f07dca907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.664 7f07dca907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.665 7f07dca907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:46:16.665 7f07dca907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.665 7f07dca907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.665 7f07dca907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:46:16.665 7f07dca907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.665 7f07dca907c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.665 7f07dca907c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:16.665 7f07dca907c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.665 7f07dca90e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:46:16.667 7f07dca90e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:46:16.668 7f07dca90c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:46:16.669 7f07dca40540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.669 7f07dca40540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:46:16.669 7f07dca40540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.669 7f07dca40540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.669 7f07dca40540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:16.670 7f07dca40540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.749 7f07dca907c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:46:16.757 7f07dca907c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:46:16.758 7f07dc9e1040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.758 7f07dc9e1040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:16.758 7f07dc9e1040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.758 7f07dc9e1040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.758 7f07dc9e1040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:16.758 7f07dc9e1040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.759 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:16.760 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:16.760 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:16.761 7f07dca407c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.761 7f07dca407c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:16.761 7f07dca407c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.761 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:16.761 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:16.762 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:16.775 7f07dca907c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:46:16.776 7f07dca407c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.776 7f07dca407c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:16.776 7f07dca407c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.777 7f07dca407c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.777 7f07dca407c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:16.777 7f07dca407c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.799 7f07dc9e1c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.799 7f07dc9e1c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:16.799 7f07dc9e1c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.832 7f07dca907c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:46:16.833 7f07dc9e1ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.833 7f07dc9e1ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:16.833 7f07dc9e1ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.836 7f07dca907c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:46:16.836 7f07dc9e1ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.837 7f07dc9e1ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:16.837 7f07dc9e1ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.837 7f07dca907c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:46:16.837 7f07dca907c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:46:16.837 7f07dca907c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:46:16.837 7f07dc9e1e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:46:16.838 7f07dc9120e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.838 7f07dc9120e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:46:16.838 7f07dc9120e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.838 7f07dc9120e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.838 7f07dc9120e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:46:16.839 7f07dc9120e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.842 7f07dca907c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:46:16.847 7f07dc9e1e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:46:16.848 7f07dc912860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.848 7f07dc912860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:46:16.848 7f07dc912860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.878 7f07dca907c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:46:16.880 7f07de123400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.881 7f07de123400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:46:16.881 7f07de123400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:16.908 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:16.909 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:16.909 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:16.911 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:16.911 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:16.912 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:17.122 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:17.123 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:17.123 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:17.123 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:17.123 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:17.124 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:17.668 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:17.704 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:17.705 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:17.705 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:17.705 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:17.705 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:17.706 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:18.669 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:19.083 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:19.084 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:19.084 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:19.085 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:19.085 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:19.085 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:19.671 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:20.661 7f07de123400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:46:20.662 7f07de123400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:46:20.662 7f07de123400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:46:20.662 7f07de123400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:46:20.663 7f07de123400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:46:20.663 7f07de123400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:46:20.663 7f07de123400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:46:20.663 7f07de123400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:46:20.663 7f07de123400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:46:20.663 7f07de123400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:46:20.663 7f07de123400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:46:20.663 7f07de123900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:46:20.664 7f07de123900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:46:20.672 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:21.525 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:21.526 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:21.526 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:21.526 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:21.526 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:21.527 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:21.673 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:22.675 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:23.676 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:24.677 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:25.678 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:26.680 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:27.308 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:27.309 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:27.309 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:27.310 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:27.310 7f07dca40a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:27.310 7f07dca40a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:27.681 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:28.682 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:29.683 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:30.684 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:31.688 7f07dca90f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:31.735 7f07dca40a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:46:31.736 7f07dca40a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:46:31.736 7f07dca40a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:46:31.966 7f07dca907c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:46:31.966 7f07e1af2b40 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:46:31.966 7f07e1af2b40 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:46:31.966 7f07dca40860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:46:31.966 7f07dca40860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:46:31.987 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:46:31.987 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:46:31.987 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:46:31.987 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:46:31.988 7f07e1af2b40 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:46:41.820 7f23c54ecc00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:46:41.820 7f23c54ecc00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:46:41.821 7f23c54ecc00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:46:41.824 7f23c54ecc00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:46:41.825 7f23c0a6e2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:46:41.825 7f23c0a6e2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:46:41.825 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:46:41.826 7f23c0a6e7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:46:41.826 7f23c0a6e7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:46:41.826 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:46:41.826 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.827 7f23c0a6e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:46:41.827 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.827 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.827 7f23c0a6e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354001826544 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:46:41.827 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:46:41.828 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.829 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.829 7f23c0a6e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:41.829 7f23c0a6e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.829 7f23c0a6eea0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:46:41.830 7f23c0a6eea0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:46:41.831 7f23c0a6ed60 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:46:41.831 7f23c0a1e680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.831 7f23c0a1e680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:46:41.831 7f23c0a1e680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.831 7f23c0a1e680 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.832 7f23c0a1e680 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:41.832 7f23c0a1e680 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.903 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:46:41.913 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:46:41.914 7f23c09b20e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.914 7f23c09b20e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:41.914 7f23c09b20e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.914 7f23c09b20e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.915 7f23c09b20e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:41.915 7f23c09b20e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.916 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:41.917 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:41.917 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:41.919 7f23c0a1e860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.919 7f23c0a1e860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:41.919 7f23c0a1e860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.920 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:41.920 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:41.921 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:41.926 7f23c0a1e860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.926 7f23c0a1e860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:41.926 7f23c0a1e860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.927 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:46:41.928 7f23c0a1e860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.928 7f23c0a1e860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:41.928 7f23c0a1e860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.936 7f23c09b2d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.937 7f23c09b2d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:41.937 7f23c09b2d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.982 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:46:41.983 7f23c09b2c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.983 7f23c09b2c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:46:41.983 7f23c09b2c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.991 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:46:41.991 7f23c09b2c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.991 7f23c09b2c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:46:41.991 7f23c09b2c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.992 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:46:41.992 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:46:41.992 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:46:41.993 7f23c09b2f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:46:41.993 7f23c08e5220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.993 7f23c08e5220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:46:41.993 7f23c08e5220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.993 7f23c08e5220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.994 7f23c08e5220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:46:41.994 7f23c08e5220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:41.997 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:46:42.002 7f23c09b2f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:46:42.002 7f23c08e5d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:42.003 7f23c08e5d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:46:42.003 7f23c08e5d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:42.024 7f23c0a6e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:46:42.026 7f23c08e5900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:42.027 7f23c08e5900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:46:42.027 7f23c08e5900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:46:42.032 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:42.082 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:42.083 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:42.083 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:42.083 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:42.084 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:42.388 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:42.389 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:42.389 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:42.390 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:42.390 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:42.390 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:42.830 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:43.232 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:43.233 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:43.233 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:43.234 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:43.234 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:43.235 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:43.831 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:44.460 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:44.462 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:44.462 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:44.463 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:44.463 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:44.463 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:44.832 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:45.826 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:46:45.828 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:46:45.828 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:46:45.828 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:46:45.828 7f23c08e54a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:46:45.828 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:46:45.829 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:46:45.829 7f23c08e54a0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:46:45.829 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:46:45.829 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:46:45.829 7f23c08e54a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:46:45.829 7f23c20e9400 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:46:45.830 7f23c20e9400 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:46:45.833 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:46.834 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:47.503 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:47.504 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:47.504 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:47.504 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:47.505 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:47.505 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:47.835 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:48.837 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:49.838 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:50.839 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:51.840 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:52.841 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:53.842 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:54.715 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:46:54.716 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:54.716 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:46:54.717 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:54.717 7f23c0a1eb80 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:46:54.717 7f23c0a1eb80 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:46:54.843 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:55.844 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:56.845 7f23c0a1e180 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:46:56.927 7f23c0a1eb80 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:46:56.927 7f23c0a1eb80 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:46:56.928 7f23c0a1eb80 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:46:57.095 7f23c0a6e7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:46:57.095 7f23c54ecc00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:46:57.096 7f23c54ecc00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:46:57.096 7f23c0a1ea40 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:46:57.096 7f23c0a1ea40 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:46:57.110 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:46:57.111 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:46:57.112 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:46:57.112 7f23c54ecc00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:47:06.290 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:47:06.290 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:47:06.290 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:47:06.297 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:47:06.298 7fa8532b92c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:47:06.298 7fa8532b92c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:47:06.300 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:47:06.301 7fa8532b97c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:47:06.301 7fa8532b97c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:47:06.303 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:47:06.303 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.303 7fa8532b97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:47:06.303 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.303 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.303 7fa8532b97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354026301699 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.50:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:47:06.303 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.305 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.305 7fa8532b97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:06.305 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.305 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.305 7fa8532b97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:06.305 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.306 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.306 7fa8532b97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:06.307 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.307 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.307 7fa8532b97c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:06.307 7fa8532b97c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.307 7fa8532b9e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:47:06.308 7fa8532b9e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:47:06.310 7fa8532b9c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:47:06.311 7fa853269540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.312 7fa853269540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:47:06.312 7fa853269540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.312 7fa853269540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.312 7fa853269540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:06.312 7fa853269540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.469 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:47:06.485 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:47:06.491 7fa853202040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.491 7fa853202040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:06.491 7fa853202040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.491 7fa853202040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.492 7fa853202040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:06.492 7fa853202040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.493 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:06.498 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:06.499 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:06.503 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:47:06.503 7fa8532697c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.505 7fa8532697c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:06.505 7fa8532697c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.505 7fa8532697c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.505 7fa8532697c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:06.505 7fa8532697c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.505 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:06.505 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:06.520 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:06.528 7fa853202c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.528 7fa853202c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:06.528 7fa853202c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.532 7fa853202c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.533 7fa853202c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:06.533 7fa853202c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.584 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:47:06.585 7fa853202ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.585 7fa853202ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:06.585 7fa853202ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.588 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:47:06.588 7fa853202ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.589 7fa853202ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:06.589 7fa853202ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.589 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:47:06.589 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:47:06.589 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:47:06.590 7fa853202e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:47:06.590 7fa85494c0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.590 7fa85494c0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:06.591 7fa85494c0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.591 7fa85494c0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.591 7fa85494c0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:06.591 7fa85494c0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.594 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:47:06.599 7fa853202e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:47:06.599 7fa85494cae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.599 7fa85494cae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:06.599 7fa85494cae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.642 7fa8532b97c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:47:06.646 7fa85313f400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.647 7fa85313f400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:47:06.648 7fa85313f400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:06.655 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:06.655 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:06.656 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:06.656 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:06.656 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:06.658 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:06.851 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:06.852 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:06.852 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:06.853 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:06.853 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:06.853 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:07.310 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:07.386 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:07.387 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:07.387 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:07.388 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:07.388 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:07.388 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:08.120 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:08.121 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:08.121 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:08.122 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:08.122 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:08.122 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:08.311 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:09.301 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:09.302 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:09.302 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:09.302 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:09.302 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:09.303 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:09.312 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:10.302 7fa85313f400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:47:10.303 7fa85313f400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:47:10.303 7fa85313f400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:47:10.303 7fa85313f400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:47:10.303 7fa85313f400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:47:10.303 7fa85313f400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:47:10.303 7fa85313f400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:47:10.303 7fa85313f400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:47:10.304 7fa85313f400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:47:10.304 7fa85313f400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:47:10.304 7fa85313f400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:47:10.304 7fa85313f900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:47:10.305 7fa85313f900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:47:10.313 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:11.314 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:11.672 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:11.673 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:11.673 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:11.673 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:11.674 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:11.674 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:12.316 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:13.317 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:14.318 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:15.319 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:16.321 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:17.321 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:17.433 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:17.434 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:17.434 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:17.435 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:17.435 7fa853269a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:17.435 7fa853269a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:18.323 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:19.324 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:20.325 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:21.326 7fa8532b9f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:21.354 7fa853269a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:47:21.354 7fa853269a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:47:21.355 7fa853269a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:47:21.681 7fa8532b97c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:47:21.681 7fa857d6c2c0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:47:21.681 7fa857d6c2c0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:47:21.682 7fa853269680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:47:21.682 7fa853269680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:47:21.696 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:47:21.697 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:47:21.698 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:47:21.698 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:47:21.698 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:47:21.698 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:47:21.698 7fa857d6c2c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:47:31.035 7f1b18cfa780 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:47:31.035 7f1b18cfa780 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:47:31.035 7f1b18cfa780 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:47:31.040 7f1b18cfa780 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:47:31.041 7f1b142722c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:47:31.041 7f1b142722c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:47:31.041 7f1b142727c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:47:31.042 7f1b142727c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:47:31.042 7f1b142727c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:47:31.042 7f1b142727c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:47:31.042 7f1b142727c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.044 7f1b142727c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:47:31.044 7f1b142727c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.044 7f1b142727c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.044 7f1b142727c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354051042376 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:47:31.044 7f1b142727c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.045 7f1b142727c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:31.046 7f1b142727c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.047 7f1b14272e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:47:31.047 7f1b14272e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:47:31.048 7f1b14272c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:47:31.049 7f1b14222540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.049 7f1b14222540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:47:31.049 7f1b14222540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.049 7f1b14222540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.049 7f1b14222540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:31.049 7f1b14222540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.124 7f1b142727c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:47:31.135 7f1b142727c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:47:31.136 7f1b140ed040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.136 7f1b140ed040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:31.136 7f1b140ed040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.136 7f1b140ed040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.137 7f1b140ed040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:31.137 7f1b140ed040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.138 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:31.138 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:31.139 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:31.140 7f1b142227c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.140 7f1b142227c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:31.140 7f1b142227c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.141 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:31.141 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:31.141 7f1b142727c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:47:31.142 7f1b142227c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.142 7f1b142227c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:31.142 7f1b142227c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.142 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:31.175 7f1b140edc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.175 7f1b140edc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:31.177 7f1b140edc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.180 7f1b140edc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.181 7f1b140edc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:31.181 7f1b140edc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.216 7f1b142727c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:47:31.217 7f1b140edae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.217 7f1b140edae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:31.217 7f1b140edae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.220 7f1b142727c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:47:31.221 7f1b140edae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.221 7f1b140edae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:31.221 7f1b140edae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.222 7f1b142727c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:47:31.222 7f1b142727c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:47:31.222 7f1b142727c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:47:31.222 7f1b140ede00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:47:31.223 7f1b158470e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.223 7f1b158470e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:31.223 7f1b158470e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.223 7f1b158470e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.224 7f1b158470e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:31.224 7f1b158470e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.227 7f1b142727c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:47:31.236 7f1b140ede00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:47:31.237 7f1b15847860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.237 7f1b15847860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:31.237 7f1b15847860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.283 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:31.285 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:31.285 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:31.289 7f1b142727c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:47:31.290 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:31.290 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:31.292 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:31.299 7f1b15822400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.300 7f1b15822400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:47:31.300 7f1b15822400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:31.578 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:31.578 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:31.578 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:31.579 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:31.579 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:31.579 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:32.048 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:32.260 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:32.261 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:32.261 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:32.261 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:32.262 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:32.262 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:33.049 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:33.541 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:33.542 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:33.542 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:33.542 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:33.542 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:33.543 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:34.050 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:35.042 7f1b15822400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:47:35.044 7f1b15822400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:47:35.044 7f1b15822400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:47:35.044 7f1b15822400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:47:35.044 7f1b15822400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:47:35.044 7f1b15822400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:47:35.044 7f1b15822400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:47:35.045 7f1b15822400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:47:35.045 7f1b15822400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:47:35.045 7f1b15822400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:47:35.045 7f1b15822400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:47:35.045 7f1b15822900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:47:35.046 7f1b15822900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:47:35.051 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:36.052 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:36.830 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:36.831 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:36.831 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:36.832 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:36.832 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:36.832 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:37.053 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:38.054 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:39.056 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:40.057 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:41.058 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:42.059 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:43.061 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:44.021 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:44.021 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:44.021 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:44.022 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:44.022 7f1b14222a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:44.022 7f1b14222a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:44.062 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:45.063 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:46.064 7f1b14272f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:46.133 7f1b14222a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:47:46.133 7f1b14222a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:47:46.134 7f1b14222a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:47:46.342 7f1b142727c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:47:46.342 7f1b18cfa780 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:47:46.342 7f1b18cfa780 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:47:46.342 7f1b14222680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:47:46.342 7f1b14222680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:47:46.357 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:47:46.358 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:47:46.359 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:47:46.359 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:47:46.359 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:47:46.359 7f1b18cfa780 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:47:55.557 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:47:55.557 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:47:55.557 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:47:55.561 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:47:55.561 7f1ebd6462c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:47:55.561 7f1ebd6462c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:47:55.562 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:47:55.562 7f1ebd6467c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:47:55.562 7f1ebd6467c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:47:55.562 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:47:55.563 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.563 7f1ebd6467c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:47:55.563 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.563 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.563 7f1ebd6467c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354075562764 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:47:55.563 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.564 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.565 7f1ebd6467c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:55.565 7f1ebd6467c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.565 7f1ebd646e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:47:55.565 7f1ebd646e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:47:55.566 7f1ebd646c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:47:55.567 7f1ebd5f6540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.567 7f1ebd5f6540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:47:55.567 7f1ebd5f6540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.567 7f1ebd5f6540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.567 7f1ebd5f6540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:55.567 7f1ebd5f6540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.626 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:47:55.634 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:47:55.635 7f1ebf756040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.635 7f1ebf756040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:55.635 7f1ebf756040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.635 7f1ebf756040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.635 7f1ebf756040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:55.635 7f1ebf756040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.637 7f1ebd5f67c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.637 7f1ebd5f67c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:55.638 7f1ebd5f67c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.638 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:55.638 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:55.638 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:55.639 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:55.639 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:55.640 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:55.640 7f1ebd5f67c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.641 7f1ebd5f67c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:55.641 7f1ebd5f67c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.642 7f1ebd5f67c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.642 7f1ebd5f67c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:55.642 7f1ebd5f67c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.650 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:47:55.650 7f1ebd5f67c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.651 7f1ebd5f67c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:55.651 7f1ebd5f67c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.699 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:47:55.699 7f1ebf756c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.700 7f1ebf756c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:47:55.701 7f1ebf756c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.704 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:47:55.704 7f1ebf756c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.705 7f1ebf756c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:47:55.705 7f1ebf756c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.705 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:47:55.706 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:47:55.706 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:47:55.706 7f1ebf756f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:47:55.707 7f1ebd5cf220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.707 7f1ebd5cf220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:55.707 7f1ebd5cf220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.707 7f1ebd5cf220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.707 7f1ebd5cf220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:55.708 7f1ebd5cf220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.711 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:47:55.720 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:55.721 7f1ebf756f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:47:55.722 7f1ebd5cf9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.722 7f1ebd5cf9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:47:55.722 7f1ebd5cf9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.722 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:55.722 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:55.723 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:55.723 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:55.724 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:55.751 7f1ebd6467c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:47:55.753 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.753 7f1ebd5cf900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:47:55.753 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:47:55.849 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:55.850 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:55.850 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:55.851 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:55.851 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:55.851 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:56.077 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:56.078 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:56.078 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:56.078 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:56.078 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:56.079 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:56.506 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:56.507 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:56.507 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:56.508 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:56.508 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:56.508 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:56.567 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:57.511 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:47:57.511 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:57.512 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:47:57.512 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:57.512 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:47:57.513 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:47:57.568 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:58.569 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:47:59.562 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:47:59.564 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:47:59.564 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:47:59.564 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:47:59.564 7f1ebd5cf900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:47:59.564 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:47:59.565 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:47:59.565 7f1ebd5cf900 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:47:59.565 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:47:59.565 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:47:59.565 7f1ebd5cf900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:47:59.566 7f1ebcf5a180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:47:59.567 7f1ebcf5a180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:47:59.570 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:00.264 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:00.264 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:00.264 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:00.265 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:00.266 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:00.267 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:00.571 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:01.572 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:02.574 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:03.575 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:04.576 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:05.577 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:06.578 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:07.579 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:07.767 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:07.769 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:07.769 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:07.769 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:07.769 7f1ebd5f6a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:07.770 7f1ebd5f6a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:08.580 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:09.581 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:10.582 7f1ebd646f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:10.582 7f1ebd5f6a40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:48:10.582 7f1ebd5f6a40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:48:10.583 7f1ebd5f6a40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:48:10.817 7f1ebd6467c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:48:10.818 7f1ec20cdfc0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:48:10.818 7f1ec20cdfc0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:48:10.818 7f1ebd5f69a0 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:48:10.818 7f1ebd5f69a0 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:48:10.831 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:48:10.832 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:48:10.832 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:48:10.832 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:48:10.832 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:48:10.832 7f1ec20cdfc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:48:19.921 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:48:19.921 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:48:19.922 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:48:19.925 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:48:19.925 7f1bbfefc2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:48:19.925 7f1bbfefc2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:48:19.925 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:48:19.926 7f1bbfefc7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:48:19.926 7f1bbfefc7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:48:19.926 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:48:19.926 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.926 7f1bbfefc7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:48:19.927 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.927 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.927 7f1bbfefc7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354099926548 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.50:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:48:19.927 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:19.928 7f1bbfefc7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.929 7f1bbfefce00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:48:19.929 7f1bbfefce00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:48:19.930 7f1bbfefcc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:48:19.930 7f1bbfeac540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.931 7f1bbfeac540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:48:19.931 7f1bbfeac540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.931 7f1bbfeac540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.931 7f1bbfeac540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:19.931 7f1bbfeac540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.988 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:48:19.996 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:48:19.997 7f1bbfe08040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.997 7f1bbfe08040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:48:19.997 7f1bbfe08040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.997 7f1bbfe08040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.998 7f1bbfe08040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:48:19.998 7f1bbfe08040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:19.999 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:19.999 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.000 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:20.001 7f1bbfeac7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.001 7f1bbfeac7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:48:20.002 7f1bbfeac7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.002 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.002 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:20.003 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:48:20.003 7f1bbfeac7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.003 7f1bbfeac7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:20.003 7f1bbfeac7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.003 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.012 7f1bbfe08c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.012 7f1bbfe08c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:20.013 7f1bbfe08c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.014 7f1bbfe08c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.014 7f1bbfe08c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:20.014 7f1bbfe08c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.038 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:48:20.039 7f1bbfe08ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.039 7f1bbfe08ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:48:20.040 7f1bbfe08ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.045 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:48:20.045 7f1bbfe08ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.045 7f1bbfe08ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:20.045 7f1bbfe08ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.046 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:48:20.046 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:48:20.046 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:48:20.046 7f1bbfe08e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:48:20.047 7f1bbfd810e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.047 7f1bbfd810e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:48:20.047 7f1bbfd810e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.047 7f1bbfd810e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.048 7f1bbfd810e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:48:20.048 7f1bbfd810e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.051 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:48:20.057 7f1bbfe08e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:48:20.058 7f1bbfd819a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.058 7f1bbfd819a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:48:20.058 7f1bbfd819a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.086 7f1bbfefc7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:48:20.088 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.088 7f1bbfd817c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:48:20.088 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:20.094 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:20.095 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.095 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:20.096 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.096 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:20.096 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.219 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:20.220 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.221 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:20.221 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.221 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:20.222 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.415 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:20.422 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.423 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:20.425 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.425 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:20.426 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.898 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:20.899 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.899 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:20.899 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.900 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:20.900 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:20.930 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:21.932 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:22.124 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:22.125 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:22.125 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:22.126 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:22.128 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:22.129 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:22.933 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:23.926 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:48:23.928 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:48:23.928 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:48:23.928 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:48:23.928 7f1bbfd817c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:48:23.929 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:48:23.929 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:48:23.929 7f1bbfd817c0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:48:23.929 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:48:23.929 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:48:23.929 7f1bbfd817c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:48:23.930 7f1bc14c3180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:48:23.932 7f1bc14c3180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:48:23.934 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:24.715 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:24.716 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:24.716 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:24.716 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:24.716 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:24.717 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:24.935 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:25.936 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:26.937 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:27.938 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:28.268 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:28.269 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:28.269 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:28.270 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:28.270 7f1bbfeaca40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:28.270 7f1bbfeaca40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:28.939 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:29.940 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:30.941 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:31.943 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:32.944 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:33.945 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:34.947 7f1bbfefcf40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:35.002 7f1bbfeaca40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:48:35.003 7f1bbfeaca40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:48:35.003 7f1bbfeaca40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:48:35.151 7f1bbfefc7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:48:35.152 7f1bc4f4e9c0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:48:35.152 7f1bc4f4e9c0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:48:35.159 7f1bbfeac860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:48:35.159 7f1bbfeac860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:48:35.175 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:48:35.175 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:48:35.175 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:48:35.176 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:48:35.177 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:48:35.177 7f1bc4f4e9c0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:48:44.197 7f945888c500 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:48:44.197 7f945888c500 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:48:44.197 7f945888c500 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:48:44.200 7f945888c500 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:48:44.200 7f9453ad42c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:48:44.201 7f9453ad42c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:48:44.201 7f9453ad47c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:48:44.201 7f9453ad47c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:48:44.202 7f9453ad47c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:48:44.202 7f9453ad47c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:48:44.202 7f9453ad47c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.202 7f9453ad47c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:48:44.202 7f9453ad47c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.202 7f9453ad47c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.202 7f9453ad47c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354124202103 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.189:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:48:44.202 7f9453ad47c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.203 7f9453ad4900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.203 7f9453ad4900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:48:44.203 7f9453ad4900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.204 7f9453ad4900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.204 7f9453ad4900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:48:44.204 7f9453ad4900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.204 7f9453ad4900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.204 7f9453ad4900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:48:44.204 7f9453ad4900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.204 7f9453ad4900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.205 7f9453ad4900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:44.205 7f9453ad4900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.205 7f9453ad4f40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:48:44.205 7f9453ad4f40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:48:44.206 7f9453ad4e00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:48:44.207 7f9453a847c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.207 7f9453a847c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:48:44.207 7f9453a847c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.207 7f9453a847c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.207 7f9453a847c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:44.207 7f9453a847c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.302 7f9453ad4900 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:48:44.317 7f9453ad4900 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:48:44.318 7f9453935220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.319 7f9453935220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:48:44.319 7f9453935220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.319 7f9453935220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.319 7f9453935220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:48:44.319 7f9453935220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.320 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:44.323 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:44.324 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:44.325 7f9453a849a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.326 7f9453a849a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:48:44.326 7f9453a849a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.326 7f9453ad4900 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:48:44.326 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:44.326 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:44.327 7f9453a849a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.327 7f9453a849a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:44.327 7f9453a849a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.328 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:44.353 7f9453935ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.353 7f9453935ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:44.353 7f9453935ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.354 7f9453935ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.355 7f9453935ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:44.355 7f9453935ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.408 7f9453ad4900 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:48:44.409 7f9453935d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.409 7f9453935d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:48:44.409 7f9453935d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.413 7f9453ad4900 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:48:44.414 7f9453935d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.414 7f9453935d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:48:44.414 7f9453935d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.414 7f9453ad4900 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:48:44.415 7f9453ad4900 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:48:44.415 7f9453ad4900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:48:44.415 7f9453935d60 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:48:44.415 7f94550ad360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.416 7f94550ad360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:48:44.416 7f94550ad360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.416 7f94550ad360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.416 7f94550ad360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:48:44.416 7f94550ad360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.419 7f9453ad4900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:48:44.423 7f9453935d60 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:48:44.423 7f94550add60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.423 7f94550add60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:48:44.423 7f94550add60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.453 7f9453ad4900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:48:44.455 7f9455097540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.456 7f9455097540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:48:44.456 7f9455097540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:48:44.457 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:44.458 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:44.458 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:44.459 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:44.459 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:44.460 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:44.743 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:44.744 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:44.744 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:44.744 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:44.744 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:44.745 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:45.206 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:45.266 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:45.266 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:45.266 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:45.267 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:45.267 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:45.267 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:46.166 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:46.167 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:46.167 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:46.167 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:46.167 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:46.168 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:46.207 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:47.209 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:47.251 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:47.252 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:47.252 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:47.252 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:47.252 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:47.253 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:48.201 7f9455097540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:48:48.203 7f9455097540 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:48:48.203 7f9455097540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:48:48.203 7f9455097540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:48:48.203 7f9455097540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:48:48.203 7f9455097540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:48:48.203 7f9455097540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:48:48.203 7f9455097540 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:48:48.204 7f9455097540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:48:48.204 7f9455097540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:48:48.204 7f9455097540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:48:48.204 7f9455097a40 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:48:48.205 7f9455097a40 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:48:48.210 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:49.211 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:49.456 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:49.458 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:49.458 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:49.458 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:49.459 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:49.459 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:50.212 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:51.213 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:52.214 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:53.215 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:54.216 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:55.128 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:48:55.129 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:55.129 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:48:55.129 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:55.129 7f9453a84c20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:48:55.130 7f9453a84c20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:48:55.217 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:56.218 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:57.219 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:58.221 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:59.222 7f9453a842c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:48:59.249 7f9453a84c20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:48:59.249 7f9453a84c20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:48:59.250 7f945666d9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:48:59.544 7f9453ad4900 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:48:59.544 7f945888c500 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:48:59.545 7f945888c500 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:48:59.545 7f9453a847c0 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:48:59.545 7f9453a847c0 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:48:59.559 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:48:59.559 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:48:59.559 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:48:59.559 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:48:59.559 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:48:59.559 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:48:59.559 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:48:59.559 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:48:59.560 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:48:59.560 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:48:59.560 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:48:59.560 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:48:59.560 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:48:59.560 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:48:59.560 7f945888c500 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:49:09.084 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:49:09.084 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:49:09.084 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:49:09.088 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:49:09.088 7f8cdb54e2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:49:09.088 7f8cdb54e2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:49:09.088 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:49:09.089 7f8cdb54e7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:49:09.089 7f8cdb54e7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:49:09.089 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:49:09.089 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.089 7f8cdb54e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:49:09.089 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.089 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.090 7f8cdb54e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354149089470 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:49:09.090 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.091 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.091 7f8cdb54e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:09.091 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.091 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.091 7f8cdb54e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:09.092 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.092 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.092 7f8cdb54e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:09.092 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.092 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.092 7f8cdb54e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:09.092 7f8cdb54e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.093 7f8cdb54ef40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:49:09.093 7f8cdb54ef40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:49:09.094 7f8cdb54ed60 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:49:09.094 7f8cdb4fe7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.094 7f8cdb4fe7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:49:09.095 7f8cdb4fe7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.095 7f8cdb4fe7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.095 7f8cdb4fe7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:09.095 7f8cdb4fe7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.152 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:49:09.162 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:49:09.163 7f8cdb4920e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.163 7f8cdb4920e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:09.163 7f8cdb4920e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.163 7f8cdb4920e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.163 7f8cdb4920e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:09.163 7f8cdb4920e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.165 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:09.165 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.166 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:09.167 7f8cdb4fe9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.167 7f8cdb4fe9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:09.167 7f8cdb4fe9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.168 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.168 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:09.169 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.169 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:49:09.170 7f8cdb4fe9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.170 7f8cdb4fe9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:09.170 7f8cdb4fe9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.171 7f8cdb4fe9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.171 7f8cdb4fe9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:09.171 7f8cdb4fe9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.179 7f8cdb492d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.179 7f8cdb492d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:09.180 7f8cdb492d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.207 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:49:09.208 7f8cdb492c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.209 7f8cdb492c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:09.209 7f8cdb492c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.212 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:49:09.213 7f8cdb492c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.213 7f8cdb492c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:09.213 7f8cdb492c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.213 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:49:09.214 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:49:09.214 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:49:09.214 7f8cdb492f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:49:09.215 7f8cdb3d1220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.215 7f8cdb3d1220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:09.215 7f8cdb3d1220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.215 7f8cdb3d1220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.215 7f8cdb3d1220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:09.215 7f8cdb3d1220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.219 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:49:09.221 7f8cdb492f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:49:09.222 7f8cdb3d19a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.222 7f8cdb3d19a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:09.222 7f8cdb3d19a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.252 7f8cdb54e7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:49:09.254 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.254 7f8cdb3d17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:49:09.254 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:09.282 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:09.283 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.283 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:09.284 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.284 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:09.284 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.446 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:09.447 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.447 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:09.448 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.448 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:09.448 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.783 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:09.784 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.784 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:09.785 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:09.785 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:09.785 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:10.093 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:10.323 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:10.324 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:10.325 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:10.325 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:10.325 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:10.326 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:11.095 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:11.683 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:11.683 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:11.684 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:11.684 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:11.684 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:11.685 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:12.096 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:13.089 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:49:13.091 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:49:13.091 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:49:13.091 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:49:13.091 7f8cdb3d17c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:49:13.091 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:49:13.091 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:49:13.091 7f8cdb3d17c0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:49:13.092 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:49:13.092 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:49:13.092 7f8cdb3d17c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:49:13.092 7f8cdcbd2180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:49:13.093 7f8cdcbd2180 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:49:13.097 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:14.040 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:14.041 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:14.041 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:14.041 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:14.042 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:14.042 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:14.097 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:15.099 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:16.101 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:17.101 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:18.102 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:19.104 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:19.513 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:19.514 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:19.514 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:19.515 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:19.515 7f8cdb4fec20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:19.515 7f8cdb4fec20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:20.105 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:21.106 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:22.107 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:23.108 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:24.109 7f8cdb4fe2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:24.138 7f8cdb4fec20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:49:24.138 7f8cdb4fec20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:49:24.139 7f8cdb4fec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:49:24.285 7f8cdb54e7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:49:24.285 7f8ce0314ec0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:49:24.285 7f8ce0314ec0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:49:24.286 7f8cdb4feb80 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:49:24.286 7f8cdb4feb80 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:49:24.299 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:49:24.300 7f8ce0314ec0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:49:33.136 7f294194fcc0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:49:33.136 7f294194fcc0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:49:33.136 7f294194fcc0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:49:33.140 7f294194fcc0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:49:33.141 7f293cecb2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:49:33.141 7f293cecb2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:49:33.141 7f293cecb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:49:33.142 7f293cecb7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:49:33.142 7f293cecb7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:49:33.142 7f293cecb7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:49:33.142 7f293cecb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.142 7f293cecb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:49:33.142 7f293cecb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.142 7f293cecb7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.142 7f293cecb7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354173142352 collector_name = collector_ip = collector_list = [ 10.0.0.189:8086, 10.0.0.38:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:49:33.143 7f293cecb7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.143 7f293cecb900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.144 7f293cecb900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:33.145 7f293cecb900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.145 7f293cecbf40 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:49:33.145 7f293cecbf40 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:49:33.146 7f293cecbe00 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:49:33.147 7f293ce7b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.147 7f293ce7b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:49:33.147 7f293ce7b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.147 7f293ce7b7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.147 7f293ce7b7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:33.147 7f293ce7b7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.208 7f293cecb900 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:49:33.216 7f293cecb900 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:49:33.217 7f293cdd40e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.218 7f293cdd40e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:33.218 7f293cdd40e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.218 7f293cdd40e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.218 7f293cdd40e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:33.218 7f293cdd40e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.219 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:33.220 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.220 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:33.222 7f293ce7b9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.222 7f293ce7b9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:33.222 7f293ce7b9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.222 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.222 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:33.223 7f293cecb900 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:49:33.223 7f293ce7b9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.224 7f293ce7b9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:33.224 7f293ce7b9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.224 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.231 7f293cdd4d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.232 7f293cdd4d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:33.232 7f293cdd4d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.233 7f293cdd4d60 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.233 7f293cdd4d60 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:33.233 7f293cdd4d60 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.259 7f293cecb900 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:49:33.260 7f293cdd4c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.260 7f293cdd4c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:33.260 7f293cdd4c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.281 7f293cecb900 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:49:33.281 7f293cdd4c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.281 7f293cdd4c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:33.281 7f293cdd4c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.282 7f293cecb900 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:49:33.282 7f293cecb900 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:49:33.282 7f293cecb900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:49:33.282 7f293cdd4f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:49:33.283 7f293cd50220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.283 7f293cd50220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:33.283 7f293cd50220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.283 7f293cd50220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.284 7f293cd50220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:33.284 7f293cd50220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.287 7f293cecb900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:49:33.292 7f293cdd4f40 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:49:33.292 7f293cd50c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.292 7f293cd50c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:33.292 7f293cd50c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.315 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:33.316 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.316 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:33.317 7f293cecb900 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:49:33.318 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.318 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:33.320 7f293e55f400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.320 7f293e55f400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:49:33.320 7f293e55f400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:33.320 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.529 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:33.530 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.530 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:33.530 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.530 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:33.531 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.795 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:33.796 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.797 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:33.797 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:33.797 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:33.798 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:34.146 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:34.528 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:34.529 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:34.530 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:34.530 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:34.530 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:34.531 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:35.147 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:35.890 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:35.891 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:35.891 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:35.892 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:35.892 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:35.892 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:36.148 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:37.142 7f293e55f400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:49:37.144 7f293e55f400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:49:37.144 7f293e55f400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:49:37.144 7f293e55f400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:49:37.144 7f293e55f400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.189:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:49:37.144 7f293e55f400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:49:37.144 7f293e55f400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:49:37.145 7f293e55f400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:49:37.145 7f293e55f400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:49:37.145 7f293e55f400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:49:37.145 7f293e55f400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:49:37.145 7f293e55f900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:49:37.146 7f293e55f900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:49:37.149 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:38.150 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:38.438 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:38.439 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:38.439 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:38.440 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:38.440 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:38.440 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:39.151 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:40.152 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:41.153 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:42.154 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:43.156 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:44.157 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:44.239 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:44.239 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:44.240 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:44.240 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:44.240 7f293ce7bc20 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:44.241 7f293ce7bc20 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:45.158 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:46.160 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:47.161 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:48.159 7f293ce7bc20 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:49:48.159 7f293ce7bc20 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:49:48.160 7f293ce7bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:49:48.162 7f293ce7b2c0 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:48.410 7f293cecb900 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:49:48.410 7f294194fcc0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:49:48.410 7f294194fcc0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:49:48.411 7f293ce7b860 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:49:48.411 7f293ce7b860 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:49:48.469 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:49:48.470 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:49:48.470 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:49:48.470 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:49:48.470 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:49:48.470 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:49:48.470 7f294194fcc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:49:57.764 7f53b277dbc0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:49:57.765 7f53b277dbc0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:49:57.765 7f53b277dbc0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:49:57.768 7f53b277dbc0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:49:57.769 7f53adcee2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:49:57.769 7f53adcee2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:49:57.769 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:49:57.769 7f53adcee7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:49:57.770 7f53adcee7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:49:57.770 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:49:57.770 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.770 7f53adcee7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:49:57.770 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.770 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.770 7f53adcee7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354197769961 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.189:8086, 10.0.0.38:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:49:57.770 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.771 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.771 7f53adcee7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:57.771 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.771 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.771 7f53adcee7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:57.772 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.772 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.772 7f53adcee7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:49:57.772 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.772 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.772 7f53adcee7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:57.772 7f53adcee7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.772 7f53adceee00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:49:57.773 7f53adceee00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:49:57.773 7f53adceec20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:49:57.774 7f53adc9e540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.774 7f53adc9e540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:49:57.774 7f53adc9e540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.774 7f53adc9e540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.774 7f53adc9e540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:57.774 7f53adc9e540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.837 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:49:57.864 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:49:57.865 7f53adc3b040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.868 7f53adc3b040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:57.868 7f53adc3b040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.868 7f53adc3b040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.868 7f53adc3b040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:57.868 7f53adc3b040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.869 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:57.870 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:57.870 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:57.871 7f53adc9e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.871 7f53adc9e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:57.871 7f53adc9e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.871 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:57.871 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:57.872 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:57.873 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:49:57.873 7f53adc9e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.873 7f53adc9e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:57.873 7f53adc9e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.874 7f53adc3b040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.875 7f53adc3b040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:57.875 7f53adc3b040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.882 7f53adc3bc20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.883 7f53adc3bc20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:57.883 7f53adc3bc20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.912 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:49:57.912 7f53adc3bae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.913 7f53adc3bae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:49:57.913 7f53adc3bae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.916 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:49:57.916 7f53adc3bae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.917 7f53adc3bae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:49:57.917 7f53adc3bae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.917 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:49:57.917 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:49:57.917 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:49:57.918 7f53adc3be00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:49:57.918 7f53adb670e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.918 7f53adb670e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:57.918 7f53adb670e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.918 7f53adb670e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.919 7f53adb670e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:57.919 7f53adb670e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.921 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:49:57.925 7f53adc3be00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:49:57.926 7f53adb67ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.926 7f53adb67ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:49:57.926 7f53adb67ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.959 7f53adcee7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:49:57.964 7f53af36b400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.965 7f53af36b400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:49:57.965 7f53af36b400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:49:57.969 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:57.970 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:57.970 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:57.972 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:57.972 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:57.973 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:58.216 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:58.217 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:58.217 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:58.218 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:58.218 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:58.218 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:58.773 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:58.797 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:58.799 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:58.799 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:58.800 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:58.800 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:58.800 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:59.774 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:49:59.805 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:49:59.807 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:59.807 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:49:59.807 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:49:59.808 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:49:59.808 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:00.776 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:01.769 7f53af36b400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:50:01.771 7f53af36b400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:50:01.771 7f53af36b400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:50:01.771 7f53af36b400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:50:01.771 7f53af36b400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:50:01.771 7f53af36b400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:50:01.771 7f53af36b400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:50:01.771 7f53af36b400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:50:01.772 7f53af36b400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:50:01.772 7f53af36b400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:50:01.772 7f53af36b400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:50:01.772 7f53af36b900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:50:01.773 7f53af36b900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:50:01.777 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:02.570 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:02.571 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:02.571 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:02.572 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:02.572 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:02.572 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:02.777 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:03.778 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:04.780 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:05.781 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:06.782 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:07.783 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:08.784 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:09.786 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:10.223 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:10.223 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:10.223 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:10.224 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:10.224 7f53adc9ea40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:10.225 7f53adc9ea40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:10.787 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:11.788 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:12.790 7f53adceef40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:12.838 7f53adc9ea40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:50:12.838 7f53adc9ea40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:50:12.839 7f53adc9ea40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:50:13.043 7f53adcee7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:50:13.044 7f53b277dbc0 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:50:13.044 7f53b277dbc0 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:50:13.044 7f53adc9e680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:50:13.044 7f53adc9e680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:50:13.058 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:50:13.059 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:50:13.060 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:50:13.060 7f53b277dbc0 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:50:22.464 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:50:22.464 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:50:22.464 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:50:22.468 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:50:22.469 7f1e5678d2c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:50:22.469 7f1e5678d2c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:50:22.469 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:50:22.469 7f1e5678d7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:50:22.470 7f1e5678d7c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:50:22.470 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:50:22.470 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.470 7f1e5678d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:50:22.470 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.470 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.470 7f1e5678d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354222470062 collector_name = collector_ip = collector_list = [ 10.0.0.38:8086, 10.0.0.189:8086, 10.0.0.50:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:50:22.470 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.471 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.471 7f1e5678d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:50:22.471 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.471 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.471 7f1e5678d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:50:22.471 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.471 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.472 7f1e5678d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:50:22.472 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.472 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.472 7f1e5678d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:22.472 7f1e5678d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.472 7f1e5678de00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:50:22.472 7f1e5678de00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:50:22.473 7f1e5678dc20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:50:22.474 7f1e5673d540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.474 7f1e5673d540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:50:22.474 7f1e5673d540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.474 7f1e5673d540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.474 7f1e5673d540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:22.474 7f1e5673d540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.540 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:50:22.549 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:50:22.550 7f1e56722040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.551 7f1e56722040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:50:22.551 7f1e56722040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.551 7f1e56722040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.551 7f1e56722040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:50:22.551 7f1e56722040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.552 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:22.553 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:22.553 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:22.554 7f1e5673d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.554 7f1e5673d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:50:22.554 7f1e5673d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.554 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:22.555 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:22.555 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:22.556 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:50:22.557 7f1e5673d7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.557 7f1e5673d7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:22.557 7f1e5673d7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.558 7f1e56722040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.558 7f1e56722040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:22.558 7f1e56722040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.566 7f1e56722c20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.567 7f1e56722c20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:22.567 7f1e56722c20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.591 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:50:22.593 7f1e56722ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.593 7f1e56722ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:50:22.593 7f1e56722ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.602 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:50:22.602 7f1e56722ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.602 7f1e56722ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:22.602 7f1e56722ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.603 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:50:22.603 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:50:22.603 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:50:22.603 7f1e56722e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:50:22.604 7f1e5889f0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.604 7f1e5889f0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:50:22.604 7f1e5889f0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.604 7f1e5889f0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.605 7f1e5889f0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:50:22.605 7f1e5889f0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.608 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:50:22.611 7f1e56722e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:50:22.611 7f1e5889f860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.611 7f1e5889f860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:50:22.611 7f1e5889f860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.644 7f1e5678d7c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:50:22.646 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.646 7f1e57d3e400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:50:22.646 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:22.657 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:22.658 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:22.658 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:22.659 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:22.659 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:22.659 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:22.913 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:22.914 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:22.914 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:22.915 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:22.915 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:22.916 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:23.298 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:23.299 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:23.299 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:23.299 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:23.300 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:23.300 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:23.474 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:23.789 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:23.790 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:23.790 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:23.790 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:23.790 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:23.791 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:24.475 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:24.587 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:24.588 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:24.588 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:24.588 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:24.588 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:24.589 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:25.476 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:26.470 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:50:26.471 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:50:26.471 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:50:26.472 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:50:26.472 7f1e57d3e400 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.38:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:50:26.472 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:50:26.472 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:50:26.472 7f1e57d3e400 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:50:26.472 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:50:26.472 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:50:26.472 7f1e57d3e400 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:50:26.473 7f1e57d3e900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:50:26.474 7f1e57d3e900 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:50:26.477 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:26.483 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:26.484 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:26.484 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:26.485 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:26.485 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:26.485 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:27.478 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:28.480 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:29.481 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:30.482 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:31.439 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:31.440 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:31.440 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:31.440 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:31.441 7f1e5673da40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:31.441 7f1e5673da40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:31.483 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:32.485 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:33.485 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:34.486 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:35.487 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:36.489 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:37.479 7f1e5673da40 [kazoo.client] [WARNING]: Failed connecting to Zookeeper within the connection retry policy. 09/14/2026 02:50:37.479 7f1e5673da40 [kazoo.client] [INFO]: Zookeeper session closed, state: CLOSED 09/14/2026 02:50:37.480 7f1e5673da40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:50:37.490 7f1e5678df40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:37.651 7f1e5678d7c0 [contrail-alarm-gen] [ERROR]: Exception AttributeError connecting to Config DB. Arguments: ("module 'gevent._gevent_cevent' has no attribute 'Timeout'",): Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 497, in connect self._zk_client.start() File "/usr/local/lib/python3.9/site-packages/kazoo/client.py", line 673, in start raise self.handler.timeout_exception("Connection time-out") kazoo.handlers.gevent.SequentialGeventHandler.timeout_exception During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/opserver/config_handler.py", line 59, in start self._vnc_db = VncObjectDBClient(self._cassandra_cfg['servers'], File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_object_db.py", line 20, in __init__ self._object_db = vnc_cassandra.VncCassandraClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/vnc_cassandra.py", line 107, in __init__ self._cassandra_driver = driverClass(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 187, in __init__ super(CassandraDriverCQL, self).__init__(server_list, **options) File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 167, in __init__ self._init_cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/api.py", line 191, in _init_cluster self._Init_Cluster() File "/usr/local/lib/python3.9/site-packages/cfgm_common/datastore/drivers/cassandra_cql.py", line 403, in _Init_Cluster self._zk_client = ZookeeperClient( File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 478, in __init__ self.connect() File "/usr/local/lib/python3.9/site-packages/cfgm_common/zkclient.py", line 499, in connect except gevent.event.Timeout as e: AttributeError: module 'gevent._gevent_cevent' has no attribute 'Timeout' 09/14/2026 02:50:37.651 7f1e5b7d6a00 [contrail-alarm-gen] [ERROR]: AlarmGen stopping everything 09/14/2026 02:50:37.651 7f1e5b7d6a00 [contrail-alarm-gen] [ERROR]: Stopped http server 09/14/2026 02:50:37.651 7f1e5673d680 [contrail-alarm-gen] [INFO]: Processing event[EvStop] in state[ClientInit] 09/14/2026 02:50:37.651 7f1e5673d680 [contrail-alarm-gen] [INFO]: Unconsumed event[EvStop] in state[ClientInit] 09/14/2026 02:50:37.685 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen killing 1 of 5 09/14/2026 02:50:37.685 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen joining 1 of 5 09/14/2026 02:50:37.685 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 1 of 5 09/14/2026 02:50:37.685 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen killing 2 of 5 09/14/2026 02:50:37.685 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen joining 2 of 5 09/14/2026 02:50:37.685 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 2 of 5 09/14/2026 02:50:37.685 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen killing 3 of 5 09/14/2026 02:50:37.686 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen joining 3 of 5 09/14/2026 02:50:37.686 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 3 of 5 09/14/2026 02:50:37.686 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen killing 4 of 5 09/14/2026 02:50:37.686 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen joining 4 of 5 09/14/2026 02:50:37.686 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 4 of 5 09/14/2026 02:50:37.686 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen killing 5 of 5 09/14/2026 02:50:37.686 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen joining 5 of 5 09/14/2026 02:50:37.686 7f1e5b7d6a00 [contrail-alarm-gen] [INFO]: AlarmGen stopped 5 of 5 09/14/2026 02:50:48.032 7fb98f0592c0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/14/2026 02:50:48.032 7fb98f0592c0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/14/2026 02:50:48.032 7fb98f0592c0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/14/2026 02:50:48.036 7fb98f0592c0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/14/2026 02:50:48.036 7fb98a4032c0 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/14/2026 02:50:48.036 7fb98a4032c0 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/14/2026 02:50:48.037 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/14/2026 02:50:48.037 7fb98a4037c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/14/2026 02:50:48.037 7fb98a4037c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/14/2026 02:50:48.038 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/14/2026 02:50:48.038 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.038 7fb98a4037c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/14/2026 02:50:48.038 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.038 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.040 7fb98a4037c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1789354248037840 collector_name = collector_ip = collector_list = [ 10.0.0.50:8086, 10.0.0.38:8086, 10.0.0.189:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/14/2026 02:50:48.044 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.046 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.046 7fb98a4037c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:50:48.046 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.046 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.046 7fb98a4037c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:50:48.046 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.046 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.046 7fb98a4037c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized] connection down >>, ] >> 09/14/2026 02:50:48.047 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.047 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.047 7fb98a4037c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:48.047 7fb98a4037c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.047 7fb98a403e00 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:50:48.047 7fb98a403e00 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/14/2026 02:50:48.049 7fb98a403c20 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/14/2026 02:50:48.050 7fb98a3b3540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.053 7fb98a3b3540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-6277-3.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/14/2026 02:50:48.053 7fb98a3b3540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.053 7fb98a3b3540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.053 7fb98a3b3540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:48.054 7fb98a3b3540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.179 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.38:5673//', 'pyamqp://guest:guest@10.0.0.189:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//'] 09/14/2026 02:50:48.196 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/14/2026 02:50:48.197 7fb98a34e040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.204 7fb98a34e040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:50:48.204 7fb98a34e040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.204 7fb98a34e040 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.205 7fb98a34e040 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.38:6379[Redis Instance initialized], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:50:48.205 7fb98a34e040 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.207 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:48.211 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:48.211 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:48.222 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:48.222 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:48.222 7fb98a3b37c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.222 7fb98a3b37c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:50:48.223 7fb98a3b37c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.223 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/14/2026 02:50:48.224 7fb98a3b37c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.224 7fb98a3b37c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.189:6379[Redis Instance initialized], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:48.224 7fb98a3b37c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.224 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:48.245 7fb98a34ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.246 7fb98a34ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.50:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:48.246 7fb98a34ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.250 7fb98a34ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.251 7fb98a34ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:48.251 7fb98a34ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.281 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:50:48.318 7fb98a34eae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.319 7fb98a34eae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/14/2026 02:50:48.319 7fb98a34eae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.323 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:50:48.378 7fb98a34eae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.378 7fb98a34eae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/14/2026 02:50:48.378 7fb98a34eae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.379 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/14/2026 02:50:48.379 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/14/2026 02:50:48.379 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/14/2026 02:50:48.379 7fb98a34ee00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/14/2026 02:50:48.380 7fb98a28a0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.380 7fb98a28a0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:50:48.380 7fb98a28a0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.380 7fb98a28a0e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.380 7fb98a28a0e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:50:48.380 7fb98a28a0e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.384 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/14/2026 02:50:48.387 7fb98a34ee00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/14/2026 02:50:48.388 7fb98a28a860 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.388 7fb98a28a860 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[] connection down >>, ] >> 09/14/2026 02:50:48.388 7fb98a28a860 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.427 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:48.429 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:48.430 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:48.432 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:48.432 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:48.434 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:48.435 7fb98a4037c0 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/14/2026 02:50:48.437 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.438 7fb98a28aea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:50:48.442 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/14/2026 02:50:48.699 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:48.700 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:48.700 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:48.700 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:48.701 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:48.701 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:49.048 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:49.137 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:49.138 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:49.138 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:49.139 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:49.139 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:49.139 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:50.049 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:50.195 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:50.196 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:50.196 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:50.197 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:50.197 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:50.197 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:51.050 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:52.038 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/14/2026 02:50:52.039 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Session Event: TCP Connected 09/14/2026 02:50:52.039 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/14/2026 02:50:52.039 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:50:52.039 7fb98a28aea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-6277-3. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.38:6379 server_addrs = [ 10.0.0.38:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.189:6379 server_addrs = [ 10.0.0.189:6379, ] status = Up >>, << type = Redis-UVE name = 10.0.0.50:6379 server_addrs = [ 10.0.0.50:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.38:5673, 10.0.0.189:5673, 10.0.0.50:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.38:9041, 10.0.0.189:9041, 10.0.0.50:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.38:2181, 10.0.0.189:2181, 10.0.0.50:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/14/2026 02:50:52.040 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/14/2026 02:50:52.040 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/14/2026 02:50:52.040 7fb98a28aea0 [contrail-alarm-gen] [DEBUG]: Send sandesh control message. uve type count # 7 09/14/2026 02:50:52.040 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/14/2026 02:50:52.040 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:50:52.040 7fb98a28aea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/14/2026 02:50:52.041 7fb98ba99680 [contrail-alarm-gen] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/14/2026 02:50:52.042 7fb98ba99680 [contrail-alarm-gen] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/14/2026 02:50:52.050 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:52.151 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:52.155 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:52.155 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:52.156 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:52.156 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:52.156 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:53.051 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:54.052 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:55.053 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:56.055 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:56.507 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.189(10.0.0.189):2181, use_ssl: False 09/14/2026 02:50:56.508 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:56.508 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.38(10.0.0.38):2181, use_ssl: False 09/14/2026 02:50:56.508 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:56.509 7fb98a3b3a40 [kazoo.client] [INFO]: Connecting to 10.0.0.50(10.0.0.50):2181, use_ssl: False 09/14/2026 02:50:56.509 7fb98a3b3a40 [kazoo.client] [WARNING]: Connection dropped: socket connection error: Connection refused 09/14/2026 02:50:57.056 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:58.057 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:50:59.058 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:51:00.059 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done 09/14/2026 02:51:01.061 7fb98a403f40 [contrail-alarm-gen] [INFO]: UVE Done