09/01/2025 02:00:54.523 7f520e0e73c0 [contrail-alarm-gen] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/01/2025 02:00:54.523 7f520e0e73c0 [contrail-alarm-gen] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/01/2025 02:00:54.523 7f520e0e73c0 [contrail-alarm-gen] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/01/2025 02:00:54.529 7f520e0e73c0 [contrail-alarm-gen] [INFO]: Found extensions for set() 09/01/2025 02:00:54.529 7f5209b7ad60 [contrail-alarm-gen] [ERROR]: Cannot write http_port 5995 to /tmp/contrail-alarm-gen.0.http_port 09/01/2025 02:00:54.529 7f5209b7ad60 [contrail-alarm-gen] [ERROR]: Starting Introspect on HTTP Port 5995 09/01/2025 02:00:54.529 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Processing event[EvStart] in state[none] 09/01/2025 02:00:54.530 7f5209b9ec20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:00:54.530 7f5209b9ec20 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/01/2025 02:00:54.530 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/01/2025 02:00:54.530 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.531 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 02:00:54.531 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.531 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.532 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 09/01/2025 02:00:54.532 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.533 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.534 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Initializing description = Redis Instance initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.56:6379[Redis Instance initialized] connection down >>, ] >> 09/01/2025 02:00:54.534 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.534 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.534 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.56:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 02:00:54.534 7f5209b9ec20 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.534 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:54.534 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:AlarmgenStatusTrace] map 09/01/2025 02:00:54.535 7f5209b46d60 [contrail-alarm-gen] [INFO]: Analytics Discovery zk start 09/01/2025 02:00:54.536 7f5209ae1ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.536 7f5209ae1ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:00:54.536 7f5209ae1ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.536 7f5209ae1ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.536 7f5209ae1ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.56:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 02:00:54.537 7f5209ae1ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.604 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.56:5673//'] 09/01/2025 02:00:54.611 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 02:00:54.612 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.612 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.56:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:00:54.612 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.612 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.612 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Initializing description = Redis Instance initialized >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Redis-UVE:10.0.0.56:6379[Redis Instance initialized], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:00:54.612 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.613 7f5209af3ae0 [kazoo.client] [INFO]: Connecting to 10.0.0.56(10.0.0.56):2181, use_ssl: False 09/01/2025 02:00:54.615 7f5209af30e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.615 7f5209af30e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:00:54.615 7f5209af30e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.615 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=60000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) 09/01/2025 02:00:54.617 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 02:00:54.617 7f5209af30e0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.618 7f5209af30e0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = Connection to Zookeeper initialized >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Zookeeper:AlarmGenerator[Connection to Zookeeper initialized] connection down >>, ] >> 09/01/2025 02:00:54.618 7f5209af30e0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.624 7f5209af3ae0 [kazoo.client] [INFO]: Zookeeper connection established, state: CONNECTED 09/01/2025 02:00:54.624 7f5209af3ae0 [contrail-alarm-gen] [INFO]: Analytics Discovery listen CONNECTED 09/01/2025 02:00:54.624 7f5209af3ae0 [contrail-alarm-gen] [INFO]: Analytics Discovery to publish None 09/01/2025 02:00:54.624 7f52099f6360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.624 7f52099f6360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper re-established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 02:00:54.625 7f52099f6360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.625 7f52099f6360 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.625 7f52099f6360 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 02:00:54.625 7f52099f6360 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.625 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=1): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 02:00:54.627 7f5209af3ae0 [kazoo.client] [DEBUG]: Received response(xid=1): ZnodeStat(czxid=217, mzxid=217, ctime=1756692033430, mtime=1756692033430, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=217) 09/01/2025 02:00:54.628 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=2): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=>) 09/01/2025 02:00:54.628 7f5209af3ae0 [kazoo.client] [DEBUG]: Received response(xid=2): [] 09/01/2025 02:00:54.629 7f5209b46d60 [contrail-alarm-gen] [INFO]: Analytics Discovery Watcher AlarmGenerator Children [] 09/01/2025 02:00:54.630 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=3): Exists(path='/analytics-discovery-/Collector', watcher=None) 09/01/2025 02:00:54.630 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 02:00:54.631 7f5209a47e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.631 7f5209a47e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:00:54.631 7f5209a47e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.631 7f5209af3ae0 [kazoo.client] [DEBUG]: Received response(xid=3): ZnodeStat(czxid=218, mzxid=218, ctime=1756692033437, mtime=1756692033437, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=218) 09/01/2025 02:00:54.631 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=4): GetChildren(path='/analytics-discovery-/Collector', watcher=>) 09/01/2025 02:00:54.632 7f5209af3ae0 [kazoo.client] [DEBUG]: Received response(xid=4): [] 09/01/2025 02:00:54.634 7f5209b46d60 [contrail-alarm-gen] [INFO]: Analytics Discovery AlarmGenerator reconnect 09/01/2025 02:00:54.634 7f5209b46d60 [contrail-alarm-gen] [INFO]: ensure /analytics-discovery-/AlarmGenerator 09/01/2025 02:00:54.634 7f5209b46d60 [contrail-alarm-gen] [INFO]: zk state CONNECTED (CONNECTED) 09/01/2025 02:00:54.635 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=5): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 02:00:54.636 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 02:00:54.636 7f5209a47e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.637 7f5209a47e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 02:00:54.637 7f5209a47e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.637 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: VNCCassandra started with driver cql 09/01/2025 02:00:54.637 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: initializing cassandra cluster 09/01/2025 02:00:54.637 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster before create 09/01/2025 02:00:54.637 7f5209a47e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection down 09/01/2025 02:00:54.638 7f52099f6ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.638 7f52099f6ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:00:54.638 7f52099f6ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.638 7f52099f6ea0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.639 7f52099f6ea0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:00:54.639 7f52099f6ea0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.642 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster created 09/01/2025 02:00:54.643 7f5209af3ae0 [kazoo.client] [DEBUG]: Received response(xid=5): ZnodeStat(czxid=217, mzxid=217, ctime=1756692033430, mtime=1756692033430, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=217) 09/01/2025 02:00:54.644 7f5209b46d60 [contrail-alarm-gen] [INFO]: check for /analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3993-1.-0 09/01/2025 02:00:54.644 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=6): Exists(path='/analytics-discovery-/AlarmGenerator/cn-jenkins-deploy-platform-ansible-os-3993-1.-0', watcher=None) 09/01/2025 02:00:54.645 7f5209a47e00 [contrail-alarm-gen] [WARNING]: RabbitMQ publish connection ESTABLISHED 09/01/2025 02:00:54.646 7f5209a0a220 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.646 7f5209a0a220 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:00:54.646 7f5209a0a220 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.649 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=7): Exists(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 02:00:54.652 7f5209af3ae0 [kazoo.client] [DEBUG]: Received response(xid=7): ZnodeStat(czxid=217, mzxid=217, ctime=1756692033430, mtime=1756692033430, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=217) 09/01/2025 02:00:54.652 7f5209af3ae0 [kazoo.client] [DEBUG]: Sending request(xid=8): GetChildren(path='/analytics-discovery-/AlarmGenerator', watcher=None) 09/01/2025 02:00:54.653 7f5209af3ae0 [kazoo.client] [DEBUG]: Received response(xid=8): [] 09/01/2025 02:00:54.653 7f5209b46d60 [contrail-alarm-gen] [INFO]: Discovery AG callback : dict_values([]) 09/01/2025 02:00:54.684 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 09/01/2025 02:00:54.686 7f5209a1e9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.686 7f5209a1e9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 09/01/2025 02:00:54.686 7f5209a1e9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.699 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 09/01/2025 02:00:54.699 7f5209a1e7c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.700 7f5209a1e7c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/01/2025 02:00:54.700 7f5209a1e7c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.705 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 09/01/2025 02:00:54.717 7f5209b9ec20 [contrail-alarm-gen] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 09/01/2025 02:00:54.718 7f52099dca40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.718 7f52099dca40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/01/2025 02:00:54.718 7f52099dca40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:54.740 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config config_type = global_system_config operation = UPDATE config = { uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"fq_name : ["default-global-system-config"]autonomous_system : 64512enable_4byte_as : falseconfig_version : "1.0"ibgp_auto_mesh : truemac_aging_time : 300igmp_enable : falseenable_security_policy_draft : falseid_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9146234606833845795, "uuid_lslong": 10867045081573795072}, "enable": true, "created": "2025-09-01T01:57:18.874446", "last_modified": "2025-09-01T01:58:57.212910", "description": null, "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-3993-1"], "attr": null, "uuid": "f52f0e8f-9fc6-4aa9-95d3-542a92a164d0"}] } 09/01/2025 02:00:54.741 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-compute config_type = alarm operation = UPDATE config = { uuid : "115cb4fe-f2de-490a-8662-1b54ea1e48ed"fq_name : ["default-global-system-config", "system-defined-address-mismatch-compute"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_ip_address", "operand2": {"uve_attribute": "VrouterAgent.control_ip", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1251073803591502090, "uuid_lslong": 9683332200319043821}, "enable": true, "created": "2025-09-01T01:57:30.104070", "last_modified": "2025-09-01T01:57:30.104070", "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-compute" } 09/01/2025 02:00:54.741 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-address-mismatch-control config_type = alarm operation = UPDATE config = { uuid : "31509527-15e1-47d0-964d-cb0c96706e72"fq_name : ["default-global-system-config", "system-defined-address-mismatch-control"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "not in", "operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3553504101098670032, "uuid_lslong": 10830535933819907698}, "enable": true, "created": "2025-09-01T01:57:30.193851", "last_modified": "2025-09-01T01:57:30.193851", "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-address-mismatch-control" } 09/01/2025 02:00:54.741 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bgp-connectivity config_type = alarm operation = UPDATE config = { uuid : "c6d3265b-1f49-4bbd-b9e7-ef7e8717335d"fq_name : ["default-global-system-config", "system-defined-bgp-connectivity"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_bgp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14326837012404456381, "uuid_lslong": 13395938943348192093}, "enable": true, "created": "2025-09-01T01:57:30.275956", "last_modified": "2025-09-01T01:57:30.275956", "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-bgp-connectivity" } 09/01/2025 02:00:54.742 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-bottle-request-size-limit config_type = alarm operation = UPDATE config = { uuid : "b2867004-a599-457d-a9fa-ca2dbac75af6"fq_name : ["default-global-system-config", "system-defined-bottle-request-size-limit"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["config-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "UveConfigReq.err_info.*.", "operand2": {"uve_attribute": null, "json_value": "True"}, "variables": []}]}]}id_perms : {"permissions": null, "uuid": {"uuid_mslong": 12864092550890866045, "uuid_lslong": 12248324434343451382}, "enable": null, "created": "2025-09-01T01:57:19.500783", "last_modified": "2025-09-01T01:57:19.500783", "description": "Bottle request size limit exceeded.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []} } 09/01/2025 02:00:54.742 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-elements-inconsistency config_type = alarm operation = UPDATE config = { uuid : "2a6c2e4c-e8fc-476a-8358-ee81d252e7d7"fq_name : ["default-global-system-config", "system-defined-conf-elements-inconsistency"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "size==", "operand1": "ContrailConfig.elements", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 3056869154938963818, "uuid_lslong": 9464576858265937879}, "enable": true, "created": "2025-09-01T01:57:30.054199", "last_modified": "2025-09-01T01:57:30.054199", "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-elements-inconsistency" } 09/01/2025 02:00:54.742 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-conf-incorrect config_type = alarm operation = UPDATE config = { uuid : "d2c019d3-f4d0-4dad-9629-518c8aa63400"fq_name : ["default-global-system-config", "system-defined-conf-incorrect"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "ContrailConfig", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15186166341629398445, "uuid_lslong": 10820269243797746688}, "enable": true, "created": "2025-09-01T01:57:30.367088", "last_modified": "2025-09-01T01:57:30.367088", "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-conf-incorrect" } 09/01/2025 02:00:54.742 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-core-files config_type = alarm operation = UPDATE config = { uuid : "fe5eb27e-a803-4f32-8bb7-fef697af9f46"fq_name : ["default-global-system-config", "system-defined-core-files"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "NodeStatus.all_core_file_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18329283790498910002, "uuid_lslong": 10067795827070312262}, "enable": true, "created": "2025-09-01T01:57:31.544086", "last_modified": "2025-09-01T01:57:31.544086", "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-core-files" } 09/01/2025 02:00:54.742 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-critical config_type = alarm operation = UPDATE config = { uuid : "43245ed8-0bf2-4d2a-bcc0-3f04c94d3b80"fq_name : ["default-global-system-config", "system-defined-disk-usage-critical"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": ">", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4838096181709131050, "uuid_lslong": 13600940164448598912}, "enable": true, "created": "2025-09-01T01:57:30.537335", "last_modified": "2025-09-01T01:57:30.537335", "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-critical" } 09/01/2025 02:00:54.743 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-disk-usage-high config_type = alarm operation = UPDATE config = { uuid : "56b56050-2b29-4543-8fa7-1b11e9eab7de"fq_name : ["default-global-system-config", "system-defined-disk-usage-high"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "range", "operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operand2": {"uve_attribute": null, "json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6248005955484206403, "uuid_lslong": 10351272032287242206}, "enable": true, "created": "2025-09-01T01:57:30.447796", "last_modified": "2025-09-01T01:57:30.447796", "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-disk-usage-high" } 09/01/2025 02:00:54.743 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-node-status config_type = alarm operation = UPDATE config = { uuid : "7f07ffa9-ff07-4f48-94fc-177e7d554fec"fq_name : ["default-global-system-config", "system-defined-node-status"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9153565873247047496, "uuid_lslong": 10735481443780481004}, "enable": true, "created": "2025-09-01T01:57:30.628902", "last_modified": "2025-09-01T01:57:30.628902", "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-node-status" } 09/01/2025 02:00:54.743 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-package-version-mismatch config_type = alarm operation = UPDATE config = { uuid : "f9e182fb-f082-4891-a62f-123744aa2413"fq_name : ["default-global-system-config", "system-defined-package-version-mismatch"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "NodeStatus.running_package_version", "operand2": {"uve_attribute": "NodeStatus.installed_package_version", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 18005816803787425937, "uuid_lslong": 11974809962785940499}, "enable": true, "created": "2025-09-01T01:57:31.699790", "last_modified": "2025-09-01T01:57:31.699790", "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-package-version-mismatch" } 09/01/2025 02:00:54.743 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-partial-sysinfo config_type = alarm operation = UPDATE config = { uuid : "c70e9d7c-8b02-410a-9abf-ecb18189b0c5"fq_name : ["default-global-system-config", "system-defined-partial-sysinfo"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.build_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14343575021455294730, "uuid_lslong": 11150891449519288517}, "enable": true, "created": "2025-09-01T01:57:30.714837", "last_modified": "2025-09-01T01:57:30.714837", "description": "System Info Incomplete.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-partial-sysinfo" } 09/01/2025 02:00:54.744 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks config_type = alarm operation = UPDATE config = { uuid : "a4c11095-51dc-44b5-8e1b-18d3fbd1b283"fq_name : ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["database-nodeconfig-database-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"uve_attribute": null, "json_value": "300"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11871788326234899637, "uuid_lslong": 10239805476498944643}, "enable": true, "created": "2025-09-01T01:57:31.616211", "last_modified": "2025-09-01T01:57:31.616211", "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-pending-cassandra-compaction-tasks" } 09/01/2025 02:00:54.744 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-connectivity config_type = alarm operation = UPDATE config = { uuid : "74d37966-56b4-4c48-9732-4a04d7da6446"fq_name : ["default-global-system-config", "system-defined-process-connectivity"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_status", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_status.state", "operand2": {"uve_attribute": null, "json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 8418205608933870664, "uuid_lslong": 10894851833224389702}, "enable": true, "created": "2025-09-01T01:57:30.803175", "last_modified": "2025-09-01T01:57:30.803175", "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-connectivity" } 09/01/2025 02:00:54.744 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-process-status config_type = alarm operation = UPDATE config = { uuid : "14b9b7a7-9c99-4b38-bcd8-8d265266a979"fq_name : ["default-global-system-config", "system-defined-process-status"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "NodeStatus.process_info", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "NodeStatus.process_info.process_state", "operand2": {"uve_attribute": null, "json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 1493426681964743480, "uuid_lslong": 13607781469830687097}, "enable": true, "created": "2025-09-01T01:57:30.892287", "last_modified": "2025-09-01T01:57:30.892287", "description": "Process Failure.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-process-status" } 09/01/2025 02:00:54.744 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-connectivity config_type = alarm operation = UPDATE config = { uuid : "a2515618-5dd0-44b0-95db-81848e008c45"fq_name : ["default-global-system-config", "system-defined-prouter-connectivity"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.connected_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11696224369911022768, "uuid_lslong": 10798366937868962885}, "enable": true, "created": "2025-09-01T01:57:30.980556", "last_modified": "2025-09-01T01:57:30.980556", "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-connectivity" } 09/01/2025 02:00:54.745 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-prouter-tsn-connectivity config_type = alarm operation = UPDATE config = { uuid : "6df864d4-f68f-43a3-8654-2b907e5740f6"fq_name : ["default-global-system-config", "system-defined-prouter-tsn-connectivity"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["prouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": []}, {"operation": "!=", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "ContrailConfig.elements.virtual_router_refs", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "size!=", "operand1": "ProuterData.tsn_agent_list", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": []}, {"operation": "==", "operand1": "ProuterData.gateway_mode", "operand2": {"uve_attribute": null, "json_value": "\"SERVER\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7924194410190816163, "uuid_lslong": 9679409398720970998}, "enable": true, "created": "2025-09-01T01:57:31.058068", "last_modified": "2025-09-01T01:57:31.058068", "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-prouter-tsn-connectivity" } 09/01/2025 02:00:54.745 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-storage-cluster-state config_type = alarm operation = UPDATE config = { uuid : "5fa09180-104f-468b-8f2a-619b87177afd"fq_name : ["default-global-system-config", "system-defined-storage-cluster-state"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["storage-cluster"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "StorageCluster.info_stats.status", "operand2": {"uve_attribute": null, "json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6890667409092331147, "uuid_lslong": 10316165217059830525}, "enable": true, "created": "2025-09-01T01:57:31.187380", "last_modified": "2025-09-01T01:57:31.187380", "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-storage-cluster-state" } 09/01/2025 02:00:54.745 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-interface config_type = alarm operation = UPDATE config = { uuid : "4062c0ce-0e60-4569-9c7d-c0bad36c505a"fq_name : ["default-global-system-config", "system-defined-vrouter-interface"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": ">=", "operand1": "VrouterAgent.down_interface_count", "operand2": {"uve_attribute": null, "json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4639482557382018409, "uuid_lslong": 11276380950649131098}, "enable": true, "created": "2025-09-01T01:57:31.275962", "last_modified": "2025-09-01T01:57:31.275962", "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-interface" } 09/01/2025 02:00:54.745 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down config_type = alarm operation = UPDATE config = { uuid : "37c98784-7b14-4ee3-b3b8-3dc5e2030c1c"fq_name : ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.l3mh_phy_interface_down", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4019893145475239651, "uuid_lslong": 12950168648613563420}, "enable": true, "created": "2025-09-01T01:57:31.973494", "last_modified": "2025-09-01T01:57:31.973494", "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-l3mh-physical-interface-link-down" } 09/01/2025 02:00:54.746 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "3bb0131f-d104-4f47-86f3-3a64c1f6f0a9"fq_name : ["default-global-system-config", "system-defined-vrouter-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4300958671510458183, "uuid_lslong": 9724180224848359593}, "enable": true, "created": "2025-09-01T01:57:31.788067", "last_modified": "2025-09-01T01:57:31.788067", "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-limit-exceeded" } 09/01/2025 02:00:54.746 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-vrouter-table-limit-exceeded config_type = alarm operation = UPDATE config = { uuid : "b79b1175-7b1c-407e-9321-4ac30dbcffd1"fq_name : ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["vrouter"]}alarm_severity : 0alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "VrouterAgent.res_table_limit", "operand2": {"uve_attribute": null, "json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 13230187526605258878, "uuid_lslong": 10601837199416426449}, "enable": true, "created": "2025-09-01T01:57:31.880047", "last_modified": "2025-09-01T01:57:31.880047", "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-vrouter-table-limit-exceeded" } 09/01/2025 02:00:54.746 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-close-reason config_type = alarm operation = UPDATE config = { uuid : "0d704036-6d06-4124-8c5e-a8015c938f24"fq_name : ["default-global-system-config", "system-defined-xmpp-close-reason"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["xmpp-peer"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "!=", "operand1": "XmppPeerInfoData.close_reason", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}, {"operation": "!=", "operand1": "XmppPeerInfoData.state_info.state", "operand2": {"uve_attribute": null, "json_value": "\"Established\""}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 968344522386194724, "uuid_lslong": 10114706536922320676}, "enable": true, "created": "2025-09-01T01:57:31.455463", "last_modified": "2025-09-01T01:57:31.455463", "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-close-reason" } 09/01/2025 02:00:54.746 7f5209b9ec20 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: AlarmgenConfigLog: name = default-global-system-config:system-defined-xmpp-connectivity config_type = alarm operation = UPDATE config = { uuid : "50a843c8-dff5-448c-a5a0-d45c3e9bbc71"fq_name : ["default-global-system-config", "system-defined-xmpp-connectivity"]parent_type : "global-system-config"parent_uuid : "7eedf3ea-84b7-4623-96cf-7fecf55ca500"uve_keys : {"uve_key": ["control-node"]}alarm_severity : 1alarm_rules : {"or_list": [{"and_list": [{"operation": "==", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": null, "json_value": "null"}, "variables": []}]}, {"and_list": [{"operation": "!=", "operand1": "BgpRouterState.num_up_xmpp_peer", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer", "json_value": null}, "variables": []}]}]}id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 5811969849151538316, "uuid_lslong": 11934772505184287857}, "enable": true, "created": "2025-09-01T01:57:31.369979", "last_modified": "2025-09-01T01:57:31.369979", "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}display_name : "system-defined-xmpp-connectivity" } 09/01/2025 02:00:55.536 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:56.538 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:57.540 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:58.531 7f52093d2f40 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:00:58.534 7f52093d2f40 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:00:58.534 7f52093d2f40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:00:58.535 7f52093d2f40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.535 7f52093d2f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:00:58.535 7f52093d2f40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:00:58.536 7f52093d2f40 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:00:58.537 7f52093d2f40 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:00:58.537 7f52093d2f40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.538 7f52093d2f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Redis-UVE name = 10.0.0.56:6379 server_addrs = [ 10.0.0.56:6379, ] status = Up >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:58.538 7f52093d2f40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.538 7f52093d2f40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.538 7f52093d2f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:00:58.539 7f52093d2f40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:58.542 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:59.544 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:00:59.616 7f52093d2f40 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:00:59.617 7f52093d2f40 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:00:59.617 7f52093d2f40 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:00.546 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:01.548 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:02.534 7f520ac51720 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:02.536 7f520ac51720 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:02.537 7f520ac51720 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:02.538 7f520ac51720 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.539 7f520ac51720 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:02.539 7f520ac51720 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:02.539 7f520ac51720 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:02.540 7f520ac51720 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:02.541 7f520ac51720 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.541 7f520ac51720 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:02.542 7f520ac51720 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.542 7f520ac51720 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.542 7f520ac51720 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:02.543 7f520ac51720 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:02.549 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:03.552 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:04.555 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:05.001 7f52099f6a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.56', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1.'], 'id_perms': {'created': '2025-09-01T02:01:04.990043', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:01:04.990043', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11043142725590816080, 'uuid_mslong': 5586765827790422298}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4d882deb-a0bd-411a-9941-1fccc2c79550'}, 'oper': 'CREATE', 'request-id': 'req-05faaaa3-e349-456f-a99e-6905adfb8346', 'type': 'analytics_snmp_node', 'uuid': '4d882deb-a0bd-411a-9941-1fccc2c79550'} 09/01/2025 02:01:05.002 7f52099f6a40 [contrail-alarm-gen] [ERROR]: Error in rabbitmq drainer greenlet: 'AnalyticsLogger' object has no attribute 'warn' 09/01/2025 02:01:05.002 7f52099f6a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection down 09/01/2025 02:01:05.003 7f520ac51720 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.003 7f520ac51720 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/01/2025 02:01:05.003 7f520ac51720 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.018 7f52099f6a40 [contrail-alarm-gen] [WARNING]: RabbitMQ drainer connection ESTABLISHED 09/01/2025 02:01:05.019 7f52093d2900 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.020 7f52093d2900 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:05.020 7f52093d2900 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:05.556 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:06.538 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:06.539 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:06.539 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:06.539 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.539 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:06.539 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:06.539 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:06.539 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:06.540 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.540 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:06.540 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.540 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.540 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:06.540 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:06.558 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:07.559 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:08.561 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:09.563 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:10.541 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:10.542 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:10.542 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:10.542 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.542 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:10.542 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:10.543 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:10.543 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:10.543 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.543 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:10.543 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.544 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.544 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:10.544 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:10.564 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:11.566 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:12.568 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:13.568 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:14.543 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:14.546 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:14.546 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:14.546 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.546 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:14.547 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:14.547 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:14.548 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:14.548 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.548 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:14.548 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.549 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.549 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:14.549 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:14.569 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:15.572 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:16.573 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:17.575 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:18.548 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:18.550 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:18.550 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:18.551 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.551 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:18.552 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:18.552 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:18.553 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:18.553 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.553 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:18.553 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.554 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.554 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:18.554 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:18.576 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:19.579 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:20.581 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:21.583 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:22.552 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:22.553 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:22.553 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:22.553 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.554 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:22.554 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:22.554 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:22.554 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:22.554 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.554 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:22.555 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.555 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.555 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:22.555 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:22.584 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:23.586 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:24.589 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:25.591 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:26.554 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:26.556 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:26.557 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:26.557 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.558 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:26.558 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:26.558 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:26.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:26.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.559 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:26.560 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.560 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.560 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:26.560 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:26.593 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:27.595 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:28.597 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:29.599 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:30.557 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:30.558 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:30.559 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:30.600 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:31.601 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:32.602 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:33.604 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:34.560 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:34.562 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:34.562 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:34.562 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.563 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:34.563 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:34.563 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:34.564 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:34.564 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.565 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:34.565 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.565 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.565 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:34.565 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:34.606 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:35.608 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:36.610 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:37.612 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:38.563 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:38.565 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:38.565 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:38.565 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.566 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:38.566 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:38.566 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:38.567 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:38.567 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.568 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:38.568 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.568 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.569 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:38.569 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:38.613 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:39.615 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:40.617 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:41.619 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:42.566 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:42.568 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:42.568 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:42.568 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.569 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:42.569 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:42.569 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:42.570 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:42.570 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.570 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:42.571 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.571 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.571 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:42.571 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:42.620 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:43.621 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:44.623 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:45.643 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:46.569 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:46.571 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:46.572 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:46.572 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.573 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:46.573 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:46.573 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:46.574 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:46.574 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.575 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:46.575 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.575 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.575 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:46.575 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:46.645 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:47.647 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:48.648 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:49.649 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:50.573 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:50.574 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:50.575 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:50.575 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.575 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:50.575 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:50.576 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:50.576 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:50.576 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.577 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:50.577 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.577 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.577 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:50.577 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:50.651 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:51.652 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:52.654 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:53.656 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:54.536 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:54.537 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.537 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:01:54.537 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.575 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:54.577 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:54.577 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:54.577 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.578 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:54.578 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:54.578 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:54.578 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:54.578 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.578 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:54.578 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.579 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.579 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:54.579 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:54.658 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:55.660 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:56.662 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:57.664 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:58.579 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:01:58.582 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:01:58.582 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:01:58.583 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.583 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:01:58.584 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:01:58.584 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:01:58.585 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:01:58.585 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.586 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:01:58.586 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.587 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.587 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:01:58.587 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:01:58.665 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:01:59.666 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:00.669 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:01.670 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:02.583 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:02.586 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:02.587 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:02.587 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.587 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:02.588 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:02.588 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:02.589 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:02.589 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.589 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:02.590 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.590 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.590 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:02.590 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:02.673 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:03.675 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:04.676 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:05.678 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:06.587 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:06.590 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:06.590 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:06.590 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.591 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:06.591 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:06.591 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:06.592 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:06.592 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.593 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:06.593 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.593 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.593 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:06.593 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:06.680 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:07.682 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:08.685 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:09.687 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:10.591 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:10.593 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:10.594 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:10.594 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.594 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:10.595 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:10.595 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:10.595 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:10.596 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.596 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:10.596 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.596 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.597 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:10.597 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:10.689 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:11.691 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:12.692 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:13.694 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:14.595 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:14.597 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:14.597 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:14.597 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.597 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:14.598 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:14.598 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:14.598 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:14.599 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.599 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:14.599 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.599 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.600 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:14.600 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:14.696 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:15.697 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:16.699 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:17.701 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:18.598 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:18.600 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:18.600 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:18.601 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.601 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:18.601 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:18.601 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:18.602 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:18.602 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.603 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:18.603 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.603 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.603 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:18.604 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:18.703 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:19.705 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:20.707 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:21.709 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:22.601 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:22.603 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:22.603 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:22.603 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.603 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:22.603 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:22.603 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:22.603 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:22.603 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.604 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:22.604 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.604 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.604 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:22.604 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:22.711 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:23.714 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:24.716 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:25.718 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:26.604 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:26.606 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:26.606 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:26.607 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.607 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:26.607 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:26.607 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:26.608 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:26.609 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.609 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:26.609 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.609 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.610 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:26.610 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:26.719 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:27.722 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:28.724 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:29.727 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:30.608 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:30.610 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:30.610 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:30.611 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.611 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:30.611 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:30.611 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:30.612 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:30.612 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.613 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:30.613 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.613 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.614 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:30.614 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:30.730 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:31.732 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:32.734 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:33.736 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:34.612 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:34.614 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:34.615 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:34.615 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.615 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:34.615 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:34.616 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:34.617 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:34.617 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.617 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:34.617 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.618 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.618 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:34.618 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:34.737 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:35.739 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:36.741 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:37.743 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:38.615 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:38.617 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:38.617 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:38.617 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.618 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:38.619 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:38.745 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:39.747 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:40.749 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:41.752 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:42.619 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:42.621 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:42.622 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:42.622 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.622 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:42.622 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:42.622 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:42.623 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:42.623 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.624 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:42.624 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.624 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.625 7f52093d2ae0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:42.625 7f52093d2ae0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:42.753 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:43.755 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:44.758 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:45.759 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:46.623 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:46.625 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:46.625 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:46.626 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.626 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:46.626 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:46.626 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:46.627 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:46.627 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.628 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:46.628 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.628 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.629 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:46.629 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:46.762 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:47.765 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:48.767 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:49.768 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:50.627 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:50.628 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:50.628 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:50.628 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.629 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:50.629 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:50.629 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:50.629 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:50.629 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.629 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:50.629 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.629 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.630 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:50.630 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:50.770 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:51.772 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:52.774 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:53.776 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:54.538 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:54.539 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.539 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:02:54.540 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.629 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:54.631 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:54.632 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:54.632 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.632 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:54.633 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:54.633 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:54.634 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:54.634 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.634 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:54.634 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.635 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.635 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:54.635 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:54.778 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:55.780 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:56.782 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:57.783 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:58.633 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:02:58.635 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:02:58.635 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:02:58.635 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.636 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:02:58.636 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:02:58.636 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:02:58.637 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:02:58.637 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.637 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:02:58.637 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.638 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.638 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:02:58.638 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:02:58.786 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:02:59.788 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:00.790 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:01.792 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:02.636 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:02.639 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:02.639 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:02.639 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.640 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:02.640 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:02.640 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:02.641 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:02.641 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.641 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:02.642 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.642 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.642 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:02.642 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:02.794 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:03.796 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:04.797 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:05.799 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:06.640 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:06.641 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:06.641 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:06.641 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.641 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:06.641 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:06.641 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:06.642 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:06.642 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.642 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:06.642 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.642 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.642 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:06.642 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:06.800 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:07.802 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:08.804 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:09.805 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:10.643 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:10.645 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:10.645 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:10.645 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.646 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:10.646 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:10.646 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:10.647 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:10.647 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.647 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:10.648 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.648 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.648 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:10.648 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:10.807 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:11.809 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:12.812 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:13.814 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:14.646 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:14.647 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:14.647 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:14.648 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.648 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:14.648 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:14.648 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:14.648 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:14.649 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.649 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:14.649 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.649 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.649 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:14.649 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:14.816 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:15.818 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:16.820 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:17.823 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:18.648 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:18.651 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:18.651 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:18.652 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.652 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:18.652 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:18.653 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:18.653 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:18.653 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.654 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:18.654 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.654 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.654 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:18.655 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:18.825 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:19.827 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:20.829 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:21.831 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:22.652 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:22.654 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:22.655 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:22.655 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.655 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:22.655 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:22.656 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:22.656 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:22.657 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.657 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:22.657 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.657 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.658 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:22.658 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:22.833 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:23.836 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:24.837 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:25.839 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:26.656 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:26.658 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:26.658 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:26.658 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.659 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:26.659 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:26.659 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:26.660 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:26.660 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.661 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:26.661 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.661 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.661 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:26.661 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:26.841 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:27.843 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:28.846 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:29.847 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:30.660 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:30.662 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:30.662 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:30.662 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.663 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:30.663 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:30.663 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:30.664 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:30.664 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.665 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:30.665 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.665 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.665 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:30.665 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:30.849 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:31.850 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:32.852 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:33.854 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:34.664 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:34.667 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:34.667 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:34.667 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.668 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:34.668 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:34.668 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:34.669 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:34.670 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.670 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:34.671 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.671 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.671 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:34.671 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:34.856 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:35.859 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:36.861 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:37.863 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:38.668 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:38.670 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:38.671 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:38.671 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.671 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:38.672 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:38.672 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:38.673 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:38.673 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.673 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:38.674 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.674 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.674 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:38.674 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:38.865 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:39.866 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:40.868 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:41.871 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:42.671 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:42.673 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:42.673 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:42.673 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.673 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:42.674 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:42.674 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:42.674 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:42.674 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.674 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:42.675 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.675 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.675 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:42.675 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:42.873 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:43.876 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:44.878 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:45.880 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:46.675 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:46.677 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:46.677 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:46.677 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.678 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:46.678 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:46.678 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:46.679 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:46.679 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.680 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:46.680 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.680 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.680 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:46.680 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:46.882 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:47.884 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:48.887 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:49.888 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:50.678 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:50.681 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:50.682 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:50.682 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.683 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:50.683 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:50.683 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:50.684 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:50.685 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.685 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:50.685 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.686 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.686 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:50.686 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:50.890 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:51.891 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:52.892 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:53.894 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:54.539 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:54.540 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.541 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:03:54.541 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.682 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:54.684 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:54.685 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:54.685 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.685 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:54.686 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:54.686 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:54.687 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:54.687 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.687 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:54.687 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.688 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.688 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:54.688 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:54.896 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:55.897 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:56.207 7f52099f6a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-5d92ffbe-d671-4442-9a6a-cbeb361f2ec9', 'type': 'global_system_config', 'uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500'} 09/01/2025 02:03:56.349 7f52099f6a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1'], 'obj_dict': {'display_name': 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1'], 'id_perms': {'created': '2025-09-01T02:03:56.337011', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:03:56.337011', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13614653776856471694, 'uuid_mslong': 13705629618940693837}}, 'parent_type': 'global-system-config', 'parent_uuid': '7eedf3ea-84b7-4623-96cf-7fecf55ca500', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'be342d85-dc08-4d4d-bcf0-f779fd75d88e', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.14'}, 'oper': 'CREATE', 'request-id': 'req-d72e219f-e9c4-4d0e-b3c6-3076bbe59a13', 'type': 'virtual_router', 'uuid': 'be342d85-dc08-4d4d-bcf0-f779fd75d88e'} 09/01/2025 02:03:56.735 7f52099f6a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'vhost0'], 'id_perms': {'created': '2025-09-01T02:03:56.715607', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-01T02:03:56.715607', 'permissions': {'group': 'reader', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10136671416836121550, 'uuid_mslong': 1509424230754177005}}, 'parent_type': 'virtual-router', 'parent_uuid': 'be342d85-dc08-4d4d-bcf0-f779fd75d88e', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '14f28d57-3178-4fed-8cac-b0f27d5f63ce', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:14:f2:8d:57:31']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': 'c39bc5a7-3ad4-482b-9be0-1adbb0ac1242'}]}, 'oper': 'CREATE', 'request-id': 'req-20622f07-cda6-43e3-8080-4a5fb623365b', 'type': 'virtual_machine_interface', 'uuid': '14f28d57-3178-4fed-8cac-b0f27d5f63ce'} 09/01/2025 02:03:56.772 7f52099f6a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-3993-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-20622f07-cda6-43e3-8080-4a5fb623365b', 'type': 'virtual_machine_interface', 'uuid': '14f28d57-3178-4fed-8cac-b0f27d5f63ce'} 09/01/2025 02:03:56.899 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:57.307 7f52099f6a40 [contrail-alarm-gen] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-375f7284-d086-48a3-96ae-3f89b820eabc', 'type': 'access_control_list', 'uuid': '699e2991-a098-40cb-81fd-a7e669d68ed7'} 09/01/2025 02:03:57.901 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:58.685 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:03:58.686 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:03:58.687 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:03:58.687 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.687 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:03:58.687 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:03:58.687 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:03:58.688 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:03:58.688 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.688 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:03:58.688 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.688 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.689 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:03:58.689 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:03:58.903 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:03:59.905 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:00.908 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:01.910 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:02.687 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:02.688 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:02.689 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:02.689 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.689 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:02.689 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:02.689 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:02.689 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:02.689 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.690 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:02.690 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.690 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.690 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:02.690 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:02.912 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:03.914 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:04.915 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:05.918 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:06.690 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:06.692 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:06.692 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:06.693 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.693 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:06.693 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:06.693 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:06.694 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:06.694 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.695 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:06.695 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.695 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.695 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:06.696 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:06.919 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:07.722 7f5209a1e180 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:07.920 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:08.922 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:09.924 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:10.693 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:10.695 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:10.695 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:10.695 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.695 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:10.695 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:10.695 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:10.696 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:10.696 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.696 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:10.696 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.696 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.696 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:10.696 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:10.926 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:11.928 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:12.930 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:13.932 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:14.696 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:14.698 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:14.698 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:14.699 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.699 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:14.699 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:14.699 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:14.700 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:14.700 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.701 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:14.701 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.701 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.701 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:14.702 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:14.934 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:15.937 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:16.938 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:17.940 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:18.699 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:18.702 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:18.702 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:18.702 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.703 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:18.703 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:18.703 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:18.704 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:18.704 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.705 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:18.705 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.705 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.705 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:18.705 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:18.942 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:19.944 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:20.645 7f5209a1e180 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:20.946 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:21.947 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:22.703 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:22.705 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:22.705 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:22.705 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.705 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:22.706 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:22.706 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:22.706 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:22.707 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.707 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:22.707 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.707 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.707 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:22.707 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:22.949 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:23.951 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:24.952 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:25.954 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:26.706 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:26.708 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:26.708 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:26.709 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.709 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:26.709 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:26.709 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:26.710 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:26.710 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.711 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:26.711 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.711 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.711 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:26.712 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:26.956 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:27.958 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:28.960 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:29.963 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:30.709 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:30.711 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:30.712 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:30.712 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.712 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:30.713 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:30.713 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:30.713 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:30.714 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.714 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:30.714 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.715 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.715 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:30.715 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:30.965 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:31.967 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:32.968 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:33.562 7f5209a1e180 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:33.970 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:34.712 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:34.714 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:34.714 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:34.715 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.715 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:34.715 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:34.715 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:34.716 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:34.716 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.717 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:34.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.718 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:34.718 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:34.971 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:35.973 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:36.976 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:37.978 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:38.716 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.717 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:38.718 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.718 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.718 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:38.718 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:38.979 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:39.981 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:40.983 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:41.985 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:42.717 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:42.720 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:42.721 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:42.721 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.721 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:42.722 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:42.722 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:42.723 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:42.723 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.724 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:42.724 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.724 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.725 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:42.725 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:42.987 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:43.989 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:44.990 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:45.992 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:46.722 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:46.724 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:46.725 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:46.725 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.726 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:46.726 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:46.726 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:46.727 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:46.728 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.728 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:46.729 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.729 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.729 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:46.730 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:46.845 7f5209a1e180 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:04:46.994 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:47.996 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:48.998 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:50.000 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:50.725 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:50.728 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:50.729 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:50.729 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.729 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:50.730 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:50.730 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:50.731 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:50.732 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.732 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:50.732 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.732 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:50.733 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:50.733 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:51.003 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:52.005 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:53.007 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:54.009 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:54.540 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:54.541 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.541 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:04:54.542 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.729 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:54.730 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:54.730 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:54.731 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.731 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:54.731 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:54.731 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:54.731 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:54.731 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.731 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:54.732 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.732 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:54.732 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:54.732 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:55.010 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:56.012 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:57.014 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:58.017 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:58.732 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:04:58.733 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:04:58.733 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:04:58.733 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.733 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:04:58.733 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:04:58.733 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:04:58.734 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:04:58.734 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.734 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:04:58.734 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.734 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:58.734 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:04:58.734 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:04:59.019 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:04:59.545 7f5209a1e180 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:00.021 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:01.023 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:02.026 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:02.734 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:02.736 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:02.736 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:02.736 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.736 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:02.736 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:02.736 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:02.737 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:02.737 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.737 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:02.737 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.737 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:02.737 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:02.737 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:03.028 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:04.030 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:05.032 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:06.034 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:06.737 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:06.738 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:06.738 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:06.738 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.739 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:06.739 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:06.739 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:06.739 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:06.739 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.739 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:06.740 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.740 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:06.740 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:06.740 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:07.035 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:08.036 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:09.038 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:10.040 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:10.739 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:10.741 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:10.741 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:10.741 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.741 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:10.741 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:10.741 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:10.741 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:10.742 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.742 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:10.742 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.742 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:10.742 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:10.742 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:11.042 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:12.044 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:12.836 7f5209a1e180 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:13.045 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:14.047 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:14.741 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:14.742 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:14.742 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:14.743 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:14.744 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:15.049 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:16.051 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:17.053 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:18.055 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:18.743 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:18.744 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:18.745 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:18.745 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.745 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:18.745 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:18.745 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:18.745 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:18.745 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.745 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:18.746 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.746 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:18.746 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:18.746 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:19.056 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:20.058 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:21.060 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:22.062 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:22.746 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:22.748 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:22.749 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:22.749 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.749 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:22.749 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:22.750 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:22.750 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:22.751 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.751 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:22.751 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.751 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:22.752 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:22.752 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:23.064 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:24.065 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:25.068 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:25.928 7f5209a1e180 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:26.069 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:26.749 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:26.751 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:26.752 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:26.752 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.752 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:26.753 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:26.753 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:26.754 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:26.754 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.754 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:26.754 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.755 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:26.755 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:26.755 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:27.069 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:28.071 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:29.074 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:30.077 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:30.752 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:30.754 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:30.754 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:30.754 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.754 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:30.754 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:30.754 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:30.755 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:30.755 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.755 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:30.755 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.755 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:30.755 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:30.755 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:31.078 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:32.081 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:33.083 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:34.085 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:34.755 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:34.758 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:34.758 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:34.758 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.759 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:34.759 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:34.760 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:34.761 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:34.761 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.761 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:34.762 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.762 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:34.762 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:34.762 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:35.087 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:36.088 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:37.091 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:38.092 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:38.759 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:38.764 7f5209a0a720 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:38.767 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:38.767 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:38.767 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.768 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:38.768 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:38.768 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:38.769 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:38.769 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.770 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:38.770 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.770 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:38.770 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:38.770 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:39.094 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:40.096 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:41.098 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:42.100 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:42.767 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:42.768 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:42.769 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:42.769 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.769 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:42.769 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:42.769 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:42.769 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:42.769 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.770 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:42.770 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.770 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:42.770 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:42.770 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:43.101 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:44.103 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:45.105 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:46.107 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:46.770 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:46.771 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:46.772 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:46.772 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.773 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:46.773 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:46.773 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:46.774 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:46.774 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.775 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:46.775 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.775 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:46.776 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:46.776 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:47.110 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:48.112 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:49.114 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:50.116 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:50.773 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:50.774 7f5209a1e180 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:50.774 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:50.774 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.774 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:50.774 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:50.775 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:50.775 7f5209a1e180 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:50.775 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.775 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:50.775 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.775 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:50.776 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:50.776 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:51.118 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:51.615 7f5209a1e180 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:05:52.120 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:53.122 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:54.124 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:54.535 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Starting the Kafka producer 09/01/2025 02:05:54.538 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-closed 09/01/2025 02:05:54.538 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name connections-created 09/01/2025 02:05:54.539 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name select-time 09/01/2025 02:05:54.539 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name io-time 09/01/2025 02:05:54.540 7f5209ae1a40 [kafka.client] [DEBUG]: Attempting to check version with node bootstrap-0 09/01/2025 02:05:54.540 7f5209ae1a40 [kafka.client] [DEBUG]: Initiating connection to node bootstrap-0 at 10.0.0.56:9092 09/01/2025 02:05:54.540 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent-received 09/01/2025 02:05:54.541 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-sent 09/01/2025 02:05:54.541 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes-received 09/01/2025 02:05:54.542 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name request-latency 09/01/2025 02:05:54.542 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name throttle-time 09/01/2025 02:05:54.542 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.bytes-sent 09/01/2025 02:05:54.543 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.bytes-received 09/01/2025 02:05:54.543 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.latency 09/01/2025 02:05:54.543 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-bootstrap-0.throttle 09/01/2025 02:05:54.544 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:54.545 7f5209a1e180 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.546 7f5209a1e180 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:05:54.546 7f5209a1e180 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.546 7f5209ae1a40 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:05:54.547 7f5209ae1a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:05:54.547 7f5209ae1a40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.56', 9092)]>: connecting to 10.0.0.56:9092 [('10.0.0.56', 9092) IPv4] 09/01/2025 02:05:54.548 7f5209ae1a40 [kafka.client] [DEBUG]: Timeouts: user 199.889660, metadata inf, idle connection inf, request inf 09/01/2025 02:05:54.549 7f5209ae1a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: established TCP connection 09/01/2025 02:05:54.549 7f5209ae1a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: checking broker Api Versions 09/01/2025 02:05:54.549 7f5209ae1a40 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:05:54.550 7f5209ae1a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 1 (timeout_ms 1600.0): ApiVersionsRequest_v4(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:05:54.551 7f5209ae1a40 [kafka.client] [DEBUG]: Timeouts: user 199.920177, metadata inf, idle connection inf, request 1599.180460 09/01/2025 02:05:54.565 7f5209ae1a40 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:05:54.566 7f5209ae1a40 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v4 09/01/2025 02:05:54.567 7f5209ae1a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 1 (16.29781723022461 ms): ApiVersionsResponse_v0(error_code=35, api_versions=[(api_key=18, min_version=0, max_version=3)]) 09/01/2025 02:05:54.568 7f5209ae1a40 [kafka.client] [DEBUG]: Timeouts: user 199.874640, metadata inf, idle connection 539970.000000, request inf 09/01/2025 02:05:54.569 7f5209ae1a40 [kafka.protocol.parser] [DEBUG]: Sending request ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:05:54.569 7f5209ae1a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 2 (timeout_ms 1600.0): ApiVersionsRequest_v3(client_software_name='kafka-python', client_software_version='2.1.6', _tagged_fields={}) 09/01/2025 02:05:54.571 7f5209ae1a40 [kafka.client] [DEBUG]: Timeouts: user 199.911118, metadata inf, idle connection 539967.000000, request 1599.023819 09/01/2025 02:05:54.577 7f5209ae1a40 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:05:54.577 7f5209ae1a40 [kafka.protocol.parser] [DEBUG]: Processing response ApiVersionsResponse_v3 09/01/2025 02:05:54.578 7f5209ae1a40 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 2 (8.313894271850586 ms): ApiVersionsResponse_v3(error_code=0, api_versions=[(api_key=0, min_version=0, max_version=8, _tagged_fields={}), (api_key=1, min_version=0, max_version=11, _tagged_fields={}), (api_key=2, min_version=0, max_version=5, _tagged_fields={}), (api_key=3, min_version=0, max_version=9, _tagged_fields={}), (api_key=4, min_version=0, max_version=4, _tagged_fields={}), (api_key=5, min_version=0, max_version=3, _tagged_fields={}), (api_key=6, min_version=0, max_version=6, _tagged_fields={}), (api_key=7, min_version=0, max_version=3, _tagged_fields={}), (api_key=8, min_version=0, max_version=8, _tagged_fields={}), (api_key=9, min_version=0, max_version=7, _tagged_fields={}), (api_key=10, min_version=0, max_version=3, _tagged_fields={}), (api_key=11, min_version=0, max_version=7, _tagged_fields={}), (api_key=12, min_version=0, max_version=4, _tagged_fields={}), (api_key=13, min_version=0, max_version=4, _tagged_fields={}), (api_key=14, min_version=0, max_version=5, _tagged_fields={}), (api_key=15, min_version=0, max_version=5, _tagged_fields={}), (api_key=16, min_version=0, max_version=4, _tagged_fields={}), (api_key=17, min_version=0, max_version=1, _tagged_fields={}), (api_key=18, min_version=0, max_version=3, _tagged_fields={}), (api_key=19, min_version=0, max_version=5, _tagged_fields={}), (api_key=20, min_version=0, max_version=4, _tagged_fields={}), (api_key=21, min_version=0, max_version=2, _tagged_fields={}), (api_key=22, min_version=0, max_version=3, _tagged_fields={}), (api_key=23, min_version=0, max_version=3, _tagged_fields={}), (api_key=24, min_version=0, max_version=1, _tagged_fields={}), (api_key=25, min_version=0, max_version=1, _tagged_fields={}), (api_key=26, min_version=0, max_version=1, _tagged_fields={}), (api_key=27, min_version=0, max_version=0, _tagged_fields={}), (api_key=28, min_version=0, max_version=3, _tagged_fields={}), (api_key=29, min_version=0, max_version=2, _tagged_fields={}), (api_key=30, min_version=0, max_version=2, _tagged_fields={}), (api_key=31, min_version=0, max_version=2, _tagged_fields={}), (api_key=32, min_version=0, max_version=3, _tagged_fields={}), (api_key=33, min_version=0, max_version=1, _tagged_fields={}), (api_key=34, min_version=0, max_version=1, _tagged_fields={}), (api_key=35, min_version=0, max_version=2, _tagged_fields={}), (api_key=36, min_version=0, max_version=2, _tagged_fields={}), (api_key=37, min_version=0, max_version=2, _tagged_fields={}), (api_key=38, min_version=0, max_version=2, _tagged_fields={}), (api_key=39, min_version=0, max_version=2, _tagged_fields={}), (api_key=40, min_version=0, max_version=2, _tagged_fields={}), (api_key=41, min_version=0, max_version=2, _tagged_fields={}), (api_key=42, min_version=0, max_version=2, _tagged_fields={}), (api_key=43, min_version=0, max_version=2, _tagged_fields={}), (api_key=44, min_version=0, max_version=1, _tagged_fields={}), (api_key=45, min_version=0, max_version=0, _tagged_fields={}), (api_key=46, min_version=0, max_version=0, _tagged_fields={}), (api_key=47, min_version=0, max_version=0, _tagged_fields={}), (api_key=48, min_version=0, max_version=0, _tagged_fields={}), (api_key=49, min_version=0, max_version=0, _tagged_fields={})], throttle_time_ms=0, _tagged_fields={}) 09/01/2025 02:05:54.580 7f5209ae1a40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.56', 9092)]>: Broker version identified as 2.6 09/01/2025 02:05:54.581 7f5209ae1a40 [kafka.conn] [INFO]: [IPv4 ('10.0.0.56', 9092)]>: Connection complete. 09/01/2025 02:05:54.581 7f5209ae1a40 [kafka.client] [DEBUG]: Node bootstrap-0 connected 09/01/2025 02:05:54.582 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name batch-size 09/01/2025 02:05:54.582 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name compression-rate 09/01/2025 02:05:54.583 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name queue-time 09/01/2025 02:05:54.583 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name records-per-request 09/01/2025 02:05:54.583 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name bytes 09/01/2025 02:05:54.583 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-retries 09/01/2025 02:05:54.584 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name errors 09/01/2025 02:05:54.584 7f5209ae1a40 [kafka.metrics.metrics] [DEBUG]: Added sensor with name record-size-max 09/01/2025 02:05:54.585 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Starting Kafka producer I/O thread. 09/01/2025 02:05:54.585 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) to node bootstrap-0 09/01/2025 02:05:54.585 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 02:05:54.586 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=[], allow_auto_topic_creation=True) 09/01/2025 02:05:54.586 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.829374, metadata 30000.000000, idle connection 539951.000000, request 29999.933004 09/01/2025 02:05:54.586 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Kafka producer started 09/01/2025 02:05:54.587 7f5209ae1a40 [contrail-alarm-gen] [INFO]: Initialized health-check KafkaProducer 09/01/2025 02:05:54.587 7f5209ae1a40 [contrail-alarm-gen] [INFO]: health-check KafkaProducer call returned 09/01/2025 02:05:54.587 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978313, metadata 30000.000000, idle connection 539950.000000, request 29998.433828 09/01/2025 02:05:54.610 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:05:54.611 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:05:54.611 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 3 (24.999141693115234 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[]) 09/01/2025 02:05:54.612 7f5209a1e180 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 0, groups: 0) 09/01/2025 02:05:54.612 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.943981, metadata 299999.501221, idle connection 539925.000000, request inf 09/01/2025 02:05:54.774 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:54.777 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:54.777 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:54.777 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.778 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:54.778 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:54.779 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:54.780 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:54.780 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.781 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:54.781 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.781 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:54.782 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:54.783 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:55.126 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:56.127 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:57.129 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:58.132 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:05:58.778 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:05:58.780 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:05:58.781 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:05:58.781 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.781 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:05:58.782 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:05:58.782 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:05:58.783 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:05:58.783 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.783 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:05:58.783 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.784 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:58.784 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:05:58.784 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:05:59.134 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:00.135 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:01.138 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:02.140 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:02.782 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:02.784 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:02.784 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:02.784 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.785 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:02.785 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:02.785 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:02.786 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:02.786 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.787 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:02.787 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.787 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:02.787 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:02.787 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:03.142 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:04.144 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:04.408 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:05.146 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:06.148 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:06.785 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:06.787 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:06.787 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:06.787 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.788 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:06.788 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:06.788 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:06.789 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:06.789 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.790 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:06.790 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.790 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:06.790 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:06.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:07.150 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:08.152 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:09.154 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:10.157 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:10.787 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:10.789 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:10.790 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:10.790 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.790 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:10.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:10.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:10.792 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:10.792 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.792 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:10.792 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.793 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:10.793 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:10.793 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:11.159 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:12.161 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:13.164 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:14.166 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:14.789 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:14.791 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:14.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:14.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.791 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:14.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:14.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:14.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:14.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.792 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:14.792 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.792 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:14.792 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:14.792 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:15.168 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:16.170 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:17.172 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:17.873 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:18.175 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:18.791 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:18.794 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:18.795 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:18.795 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.796 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:18.796 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:18.796 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:18.797 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:18.798 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.798 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:18.799 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.799 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:18.799 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:18.800 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:19.176 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:20.179 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:21.181 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:22.183 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:22.794 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:22.797 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:22.797 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:22.797 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.798 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:22.798 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:22.798 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:22.799 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:22.799 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.800 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:22.800 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.800 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:22.800 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:22.800 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:23.185 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:24.187 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:25.189 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:26.192 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:26.798 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:26.800 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:26.800 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:26.800 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.800 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:26.801 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:26.801 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:26.801 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:26.801 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.801 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:26.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:26.802 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:26.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:27.194 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:28.196 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:29.198 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:30.199 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:30.720 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:30.800 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:30.801 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:30.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:30.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.802 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:30.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:30.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:30.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:30.802 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.803 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:30.803 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.803 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:30.803 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:30.803 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:31.202 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:32.204 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:33.206 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:34.209 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:34.803 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:34.805 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:34.805 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:34.805 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.806 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:34.806 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:34.806 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:34.807 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:34.807 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.808 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:34.808 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.808 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:34.808 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:34.809 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:35.212 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:36.214 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:37.216 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:38.218 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:38.806 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:38.808 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:38.808 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:38.809 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.809 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:38.809 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:38.810 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:38.810 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:38.811 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.811 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:38.811 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.811 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:38.811 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:38.812 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:39.220 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:40.222 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:41.224 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:42.226 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:42.809 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:42.811 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:42.811 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:42.811 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.812 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:42.812 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:42.812 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:42.813 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:42.813 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.814 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:42.814 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.814 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:42.814 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:42.814 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:43.228 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:43.515 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:44.230 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:45.231 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:46.234 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:46.812 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:46.814 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:46.814 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:46.815 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.815 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:46.815 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:46.816 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:46.817 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:46.817 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.817 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:46.817 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.818 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:46.818 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:46.818 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:47.236 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:48.238 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:49.240 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:50.242 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:50.816 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:50.818 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:50.818 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:50.818 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.819 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:50.819 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:50.819 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:50.819 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:50.820 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.820 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:50.820 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.820 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:50.821 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:50.821 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:51.245 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:52.247 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:53.248 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:54.250 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:54.545 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:54.546 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.546 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:06:54.547 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.819 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:54.820 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:54.821 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:54.821 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.821 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:54.821 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:54.821 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:54.821 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:54.821 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.822 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:54.822 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.822 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:54.822 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:54.822 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:55.252 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:56.254 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:56.412 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:06:57.256 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:58.259 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:06:58.822 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:06:58.823 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:06:58.823 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:06:58.823 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:58.824 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:06:58.825 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:06:59.261 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:00.263 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:01.265 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:02.266 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:02.825 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:02.827 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:02.827 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:02.828 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.828 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:02.829 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:02.829 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:02.830 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:02.830 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.830 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:02.831 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.831 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:02.831 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:02.831 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:03.268 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:04.270 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:05.272 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:06.273 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:06.828 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:06.831 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:06.832 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:06.832 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.832 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:06.832 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:06.833 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:06.833 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:06.834 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.834 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:06.834 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.835 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:06.835 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:06.835 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:07.275 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:08.277 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:09.243 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:09.279 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:10.281 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:10.832 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:10.835 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:10.835 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:10.835 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.835 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:10.836 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:10.836 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:10.837 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:10.837 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.837 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:10.838 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.838 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:10.838 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:10.838 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:11.283 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:12.285 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:13.289 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:14.291 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:14.835 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:14.838 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:14.838 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:14.838 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.839 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:14.839 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:14.839 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:14.840 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:14.840 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.841 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:14.841 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.841 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:14.841 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:14.842 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:15.293 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:16.295 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:17.297 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:18.299 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:18.838 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:18.840 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:18.840 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:18.840 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.840 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:18.840 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:18.841 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:18.841 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:18.841 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.841 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:18.841 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.841 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:18.842 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:18.842 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:19.302 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:20.305 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:21.307 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:22.309 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:22.580 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:22.841 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:22.843 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:22.843 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:22.843 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.844 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:22.844 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:22.844 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:22.845 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:22.845 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.846 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:22.846 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.846 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:22.846 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:22.847 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:23.311 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:24.313 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:25.315 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:26.318 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:26.844 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:26.846 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:26.846 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:26.847 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.847 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:26.847 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:26.847 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:26.848 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:26.848 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.849 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:26.849 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.849 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:26.849 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:26.850 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:27.319 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:28.322 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:29.324 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:30.326 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:30.847 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:30.850 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:30.850 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:30.850 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.851 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:30.851 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:30.851 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:30.852 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:30.852 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.853 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:30.853 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.853 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:30.853 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:30.853 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:31.328 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:32.330 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:33.332 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:34.334 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:34.850 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:34.851 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:34.852 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:34.852 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.852 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:34.852 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:34.852 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:34.853 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:34.853 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.853 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:34.853 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.853 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:34.854 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:34.854 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:35.336 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:35.586 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:36.338 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:37.340 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:38.341 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:38.852 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:38.854 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:38.854 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:38.854 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.854 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:38.854 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:38.854 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:38.855 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:38.855 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.855 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:38.855 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.855 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:38.855 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:38.856 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:39.343 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:40.345 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:41.347 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:42.349 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:42.855 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:42.857 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:42.858 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:42.858 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.858 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:42.858 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:42.859 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:42.859 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:42.860 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.860 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:42.861 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.861 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:42.861 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:42.861 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:43.350 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:44.352 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:45.355 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:46.358 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:46.859 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:46.861 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:46.862 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:46.862 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.862 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:46.863 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:46.863 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:46.864 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:46.864 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.864 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:46.864 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.865 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:46.865 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:46.865 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:47.360 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:48.362 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:48.591 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:07:49.364 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:50.367 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:50.863 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:50.865 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:50.865 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:50.865 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.866 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:50.866 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:50.866 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:50.867 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:50.868 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.868 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:50.868 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.869 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:50.869 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:50.869 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:51.369 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:52.372 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:53.374 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:54.376 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:54.546 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:07:54.547 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.547 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:07:54.548 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.587 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:54.589 7f5209a1e180 [kafka.client] [DEBUG]: Initializing connection to node 1 for metadata request 09/01/2025 02:07:54.589 7f5209a1e180 [kafka.client] [DEBUG]: Initiating connection to node 1 at 10.0.0.56:9092 09/01/2025 02:07:54.590 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.bytes-sent 09/01/2025 02:07:54.591 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.bytes-received 09/01/2025 02:07:54.591 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.latency 09/01/2025 02:07:54.591 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name node-1.throttle 09/01/2025 02:07:54.592 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 None]>: creating new socket 09/01/2025 02:07:54.593 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: setting socket option (6, 1, 1) 09/01/2025 02:07:54.593 7f5209a1e180 [kafka.conn] [INFO]: [IPv4 ('10.0.0.56', 9092)]>: connecting to 10.0.0.56:9092 [('10.0.0.56', 9092) IPv4] 09/01/2025 02:07:54.594 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999984.773397, metadata inf, idle connection 419944.000000, request inf 09/01/2025 02:07:54.594 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: established TCP connection 09/01/2025 02:07:54.595 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: checking broker Api Versions 09/01/2025 02:07:54.595 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Using pre-configured api_version (2, 6) for ApiVersions 09/01/2025 02:07:54.595 7f5209a1e180 [kafka.conn] [INFO]: [IPv4 ('10.0.0.56', 9092)]>: Connection complete. 09/01/2025 02:07:54.595 7f5209a1e180 [kafka.client] [DEBUG]: Node 1 connected 09/01/2025 02:07:54.595 7f5209a1e180 [kafka.conn] [INFO]: [IPv4 ('10.0.0.56', 9092)]>: Closing connection. 09/01/2025 02:07:54.596 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: reconnect backoff 0.0404723611263787 after 1 failures 09/01/2025 02:07:54.597 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:54.597 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:54.597 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 1 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:54.598 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.918304, metadata 30000.000000, idle connection 419940.000000, request 29999.933004 09/01/2025 02:07:54.599 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.972591, metadata 30000.000000, idle connection 419939.000000, request 29998.905182 09/01/2025 02:07:54.688 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 1 09/01/2025 02:07:54.689 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:54.689 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 1 (91.46595001220703 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:07:54.690 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.967585, metadata 99.880371, idle connection 419847.000000, request inf 09/01/2025 02:07:54.691 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.10341763496398926 secs. 09/01/2025 02:07:54.691 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:54.692 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979267, metadata 98.276123, idle connection 419845.000000, request inf 09/01/2025 02:07:54.791 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:54.792 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:54.792 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 2 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:54.792 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.913774, metadata 30000.000000, idle connection 419745.000000, request 29999.946833 09/01/2025 02:07:54.793 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.985466, metadata 30000.000000, idle connection 419744.000000, request 29999.211788 09/01/2025 02:07:54.818 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 2 09/01/2025 02:07:54.818 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:54.819 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 2 (26.176929473876953 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:07:54.820 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.974260, metadata 99.906250, idle connection 419718.000000, request inf 09/01/2025 02:07:54.820 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.2326204776763916 secs. 09/01/2025 02:07:54.820 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:54.821 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.977360, metadata 98.367188, idle connection 419716.000000, request inf 09/01/2025 02:07:54.866 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:54.867 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:54.867 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:54.867 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.867 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:54.867 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:54.867 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:54.867 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:54.867 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.868 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:54.868 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.868 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.868 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:54.868 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:54.920 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:54.921 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:54.921 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 3 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:54.921 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.282846, metadata 30000.000000, idle connection 419616.000000, request 29999.961615 09/01/2025 02:07:54.921 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.987612, metadata 30000.000000, idle connection 419616.000000, request 29999.442577 09/01/2025 02:07:54.930 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 3 09/01/2025 02:07:54.931 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:54.931 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 3 (9.920835494995117 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:07:54.931 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.989996, metadata 99.955322, idle connection 419606.000000, request inf 09/01/2025 02:07:54.931 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.34398698806762695 secs. 09/01/2025 02:07:54.931 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:54.932 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.991188, metadata 99.364014, idle connection 419606.000000, request inf 09/01/2025 02:07:55.032 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:55.033 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.033 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 4 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.034 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.667488, metadata 30000.000000, idle connection 419504.000000, request 29999.929190 09/01/2025 02:07:55.035 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.971876, metadata 30000.000000, idle connection 419503.000000, request 29998.904943 09/01/2025 02:07:55.044 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 4 09/01/2025 02:07:55.044 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:55.045 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 4 (10.847091674804688 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:07:55.045 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.977837, metadata 99.913574, idle connection 419492.000000, request inf 09/01/2025 02:07:55.046 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.45819711685180664 secs. 09/01/2025 02:07:55.046 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:55.047 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.980221, metadata 98.614746, idle connection 419491.000000, request inf 09/01/2025 02:07:55.147 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:55.147 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.148 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 5 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.148 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.250732, metadata 30000.000000, idle connection 419389.000000, request 29999.899626 09/01/2025 02:07:55.150 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.976168, metadata 30000.000000, idle connection 419388.000000, request 29998.585939 09/01/2025 02:07:55.157 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 5 09/01/2025 02:07:55.158 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:55.158 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 5 (9.671926498413086 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:07:55.159 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978313, metadata 99.917725, idle connection 419379.000000, request inf 09/01/2025 02:07:55.159 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.5716323852539062 secs. 09/01/2025 02:07:55.159 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:55.160 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.981651, metadata 98.686523, idle connection 419377.000000, request inf 09/01/2025 02:07:55.260 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:55.261 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.261 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 6 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.261 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.498211, metadata 30000.000000, idle connection 419276.000000, request 29999.925375 09/01/2025 02:07:55.263 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978075, metadata 30000.000000, idle connection 419275.000000, request 29998.812437 09/01/2025 02:07:55.274 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 6 09/01/2025 02:07:55.275 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:55.275 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 6 (13.704299926757812 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:07:55.276 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.975929, metadata 99.840332, idle connection 419261.000000, request inf 09/01/2025 02:07:55.276 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.6890642642974854 secs. 09/01/2025 02:07:55.277 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:55.277 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979029, metadata 98.540039, idle connection 419260.000000, request inf 09/01/2025 02:07:55.377 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:55.378 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:55.378 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.379 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 7 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.379 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.289045, metadata 30000.000000, idle connection 419159.000000, request 29999.944210 09/01/2025 02:07:55.379 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.988327, metadata 30000.000000, idle connection 419158.000000, request 29999.387264 09/01/2025 02:07:55.389 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 7 09/01/2025 02:07:55.390 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:55.390 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 7 (11.64388656616211 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:07:55.391 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.976168, metadata 99.862061, idle connection 419146.000000, request inf 09/01/2025 02:07:55.392 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.8042511940002441 secs. 09/01/2025 02:07:55.392 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:55.393 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978552, metadata 98.437256, idle connection 419145.000000, request inf 09/01/2025 02:07:55.493 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:55.494 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.494 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 8 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.495 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.124371, metadata 30000.000000, idle connection 419043.000000, request 29999.894381 09/01/2025 02:07:55.496 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978552, metadata 30000.000000, idle connection 419042.000000, request 29998.780012 09/01/2025 02:07:55.506 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 8 09/01/2025 02:07:55.506 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:55.507 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 8 (12.329578399658203 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=5, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[])]) 09/01/2025 02:07:55.508 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.975452, metadata 99.885986, idle connection 419030.000000, request inf 09/01/2025 02:07:55.508 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 0.9210026264190674 secs. 09/01/2025 02:07:55.509 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Requesting metadata update for topic HEALTH_CHECK_TOPIC 09/01/2025 02:07:55.510 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979267, metadata 98.138672, idle connection 419028.000000, request inf 09/01/2025 02:07:55.610 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:07:55.610 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.611 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 9 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:07:55.611 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.694429, metadata 30000.000000, idle connection 418926.000000, request 29999.926090 09/01/2025 02:07:55.612 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.969492, metadata 30000.000000, idle connection 418925.000000, request 29998.704672 09/01/2025 02:07:55.623 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 9 09/01/2025 02:07:55.623 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:07:55.626 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 9 (14.708757400512695 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=5, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=10, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=20, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=25, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=26, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=11, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=4, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=16, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=17, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=23, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=28, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=13, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=22, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=8, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=2, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=29, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=19, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=14, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=1, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=7, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[])])]) 09/01/2025 02:07:55.629 7f5209a1e180 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 1, groups: 0) 09/01/2025 02:07:55.629 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.975214, metadata 299999.575928, idle connection 418908.000000, request inf 09/01/2025 02:07:55.629 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: _wait_on_metadata woke after 1.0418879985809326 secs. 09/01/2025 02:07:55.630 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) 09/01/2025 02:07:55.630 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) is either full or getting a new batch 09/01/2025 02:07:55.631 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:07:55.631 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:07:55.631 7f5209ae1a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) 09/01/2025 02:07:55.633 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.records-per-batch 09/01/2025 02:07:55.633 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.bytes 09/01/2025 02:07:55.634 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.compression-rate 09/01/2025 02:07:55.634 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-retries 09/01/2025 02:07:55.634 7f5209a1e180 [kafka.metrics.metrics] [DEBUG]: Added sensor with name topic.HEALTH_CHECK_TOPIC.record-errors 09/01/2025 02:07:55.635 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:07:55.635 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02lV\xb6\xaa\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x07\xee\xee\x00\x00\x01\x99\x03\x07\xee\xee\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:07:55.636 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02lV\xb6\xaa\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x07\xee\xee\x00\x00\x01\x99\x03\x07\xee\xee\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:07:55.637 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02lV\xb6\xaa\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x07\xee\xee\x00\x00\x01\x99\x03\x07\xee\xee\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:07:55.637 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 10 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02lV\xb6\xaa\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x07\xee\xee\x00\x00\x01\x99\x03\x07\xee\xee\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:07:55.638 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 299990.638428, idle connection 418900.000000, request 29999.900341 09/01/2025 02:07:55.639 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979982, metadata 299989.591797, idle connection 418898.000000, request 29998.865366 09/01/2025 02:07:55.768 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 10 09/01/2025 02:07:55.769 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:07:55.769 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 10 (131.28018379211426 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:07:55.770 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=3, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:07:55.771 7f5209a1e180 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=3) with base offset 0 log start offset 0. 09/01/2025 02:07:55.771 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.975452, metadata 299857.320312, idle connection 418766.000000, request inf 09/01/2025 02:07:55.772 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:07:56.379 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:57.381 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:58.383 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:07:58.868 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:07:58.870 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:07:58.870 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:07:58.870 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.870 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:07:58.871 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:07:58.871 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:07:58.871 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:07:58.872 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.872 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:07:58.872 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.872 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:58.872 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:07:58.873 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:07:59.385 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:00.387 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:01.389 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:01.498 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:02.391 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:02.870 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:02.872 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:02.873 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:02.873 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.873 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:02.874 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:02.874 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:02.875 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:02.875 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.875 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:02.875 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.876 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:02.876 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:02.876 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:03.393 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:04.396 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:05.399 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:06.401 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:06.873 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:06.875 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:06.876 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:06.876 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.877 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:06.877 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:06.877 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:06.878 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:06.878 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.878 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:06.879 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.879 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:06.879 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:06.879 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:07.403 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:08.406 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:09.408 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:10.411 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:10.877 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:10.880 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:10.880 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:10.880 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.881 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:10.881 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:10.881 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:10.882 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:10.882 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.882 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:10.883 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.883 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:10.883 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:10.883 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:11.413 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:12.415 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:13.418 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:14.391 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:14.419 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:14.881 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:14.883 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:14.884 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:14.884 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.885 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:14.885 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:14.886 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:14.887 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:14.887 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.887 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:14.888 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.888 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:14.888 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:14.889 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:15.421 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:16.423 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:17.425 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:18.427 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:18.884 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:18.886 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:18.887 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:18.887 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.888 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:18.888 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:18.888 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:18.889 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:18.889 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.889 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:18.890 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.890 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:18.890 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:18.890 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:19.429 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:20.431 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:21.433 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:22.436 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:22.887 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:22.890 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:22.890 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:22.890 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.891 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:22.891 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:22.891 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:22.892 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:22.892 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.892 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:22.893 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.893 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:22.893 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:22.893 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:23.438 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:24.440 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:25.441 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:26.443 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:26.890 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:26.893 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:26.893 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:26.893 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.893 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:26.893 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:26.893 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:26.894 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:26.894 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.894 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:26.894 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.895 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:26.895 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:26.895 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:27.445 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:27.823 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:28.447 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:29.448 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:30.449 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:30.893 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:30.896 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:30.896 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:30.897 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.897 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:30.897 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:30.897 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:30.898 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:30.899 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.899 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:30.899 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.900 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:30.900 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:30.900 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:31.451 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:32.453 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:33.455 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:34.457 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:34.897 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:34.898 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:34.898 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:34.898 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.899 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:34.899 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:34.899 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:34.899 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:34.899 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.899 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:34.900 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.900 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:34.900 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:34.900 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:35.459 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:36.460 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:37.462 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:38.464 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:38.899 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:38.900 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:38.900 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:38.901 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.901 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:38.901 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:38.901 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:38.902 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:38.902 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.902 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:38.902 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.902 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:38.902 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:38.902 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:39.466 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:40.468 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:40.676 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:41.471 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:42.472 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:42.901 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:42.904 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:42.904 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:42.904 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.905 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:42.905 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:42.905 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:42.907 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:42.907 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.908 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:42.908 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.908 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:42.909 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:42.909 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:43.474 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:44.476 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:45.477 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:46.479 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:46.904 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:46.907 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:46.907 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:46.907 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.908 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:46.908 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:46.908 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:46.909 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:46.909 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.909 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:46.910 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.910 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:46.910 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:46.910 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:47.480 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:48.482 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:49.484 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:50.486 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:50.908 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:50.910 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:50.910 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:50.911 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.911 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:50.911 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:50.911 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:50.912 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:50.912 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.913 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:50.913 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.913 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:50.913 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:50.914 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:51.487 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:52.489 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:53.491 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:53.879 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:08:54.493 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:54.547 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:08:54.547 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.548 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:08:54.548 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.911 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:54.913 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:54.913 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:54.913 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.913 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:54.913 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:54.914 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:54.914 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:54.914 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.914 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:54.915 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.915 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:54.915 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:54.915 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:55.496 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:56.497 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:57.498 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:58.500 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:08:58.914 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:08:58.916 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:08:58.917 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:08:58.917 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.917 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:08:58.918 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:08:58.918 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:08:58.918 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:08:58.919 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.919 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:08:58.919 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.919 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:58.920 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:08:58.920 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:08:59.502 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:00.504 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:01.506 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:02.508 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:02.917 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:02.919 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:02.920 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:02.920 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.920 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:02.921 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:02.921 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:02.922 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:02.922 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.922 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:02.923 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.923 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:02.923 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:02.923 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:03.510 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:04.512 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:05.514 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:06.515 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:06.920 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:06.922 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:06.923 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:06.923 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.924 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:06.924 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:06.924 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:06.925 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:06.925 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.925 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:06.926 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.926 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:06.926 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:06.926 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:07.050 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:07.518 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:08.520 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:09.521 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:10.523 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:10.923 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:10.926 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:10.926 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:10.926 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.927 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:10.927 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:10.927 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:10.928 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:10.929 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.929 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:10.930 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.930 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:10.930 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:10.931 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:11.526 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:12.527 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:13.530 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:14.531 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:14.926 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:14.928 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:14.929 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:14.929 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.929 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:14.929 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:14.929 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:14.930 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:14.930 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.930 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:14.931 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.931 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:14.931 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:14.931 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:15.533 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:16.536 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:17.538 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:18.540 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:18.929 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:18.931 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:18.932 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:18.932 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.932 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:18.933 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:18.933 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:18.934 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:18.934 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.934 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:18.934 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.934 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:18.935 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:18.935 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:19.542 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:20.139 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:20.544 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:21.545 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:22.547 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:22.932 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:22.935 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:22.935 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:22.935 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.936 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:22.936 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:22.936 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:22.937 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:22.937 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.938 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:22.938 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.938 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:22.938 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:22.939 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:23.549 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:24.550 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:25.552 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:26.554 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:26.936 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:26.938 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:26.938 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:26.938 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.939 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:26.939 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:26.939 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:26.940 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:26.940 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.941 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:26.941 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.941 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:26.941 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:26.942 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:27.556 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:28.558 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:29.560 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:30.562 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:30.939 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:30.941 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:30.942 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:30.942 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.942 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:30.943 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:30.943 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:30.943 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:30.944 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.944 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:30.944 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.945 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:30.945 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:30.945 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:31.564 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:32.565 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:33.399 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:33.567 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:34.569 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:34.942 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:34.944 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:34.944 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:34.944 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.944 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:34.944 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:34.945 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:34.945 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:34.945 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.945 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:34.945 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.946 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:34.946 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:34.946 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:35.572 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:36.573 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:37.575 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:38.576 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:38.945 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:38.947 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:38.947 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:38.947 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.948 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:38.948 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:38.948 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:38.949 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:38.949 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.949 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:38.950 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.950 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:38.950 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:38.950 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:39.578 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:40.579 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:41.581 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:42.582 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:42.947 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:42.949 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:42.950 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:42.950 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.950 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:42.950 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:42.950 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:42.951 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:42.951 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.951 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:42.952 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.952 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:42.952 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:42.952 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:43.584 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:44.587 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:45.589 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:46.316 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:46.591 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:46.950 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:46.953 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:46.953 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:46.954 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.954 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:46.954 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:46.954 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:46.955 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:46.955 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.956 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:46.956 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.956 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:46.957 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:46.957 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:47.592 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:48.594 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:49.596 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:50.598 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:50.954 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:50.956 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:50.956 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:50.957 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.957 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:50.957 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:50.958 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:50.958 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:50.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.959 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:50.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:50.960 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:50.960 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:51.600 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:52.602 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:53.603 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:54.547 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:09:54.548 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.549 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:09:54.549 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.604 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:54.957 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:54.958 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:54.958 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:54.958 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.959 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:54.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:54.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:54.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:54.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.959 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:54.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:54.960 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:54.960 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:55.607 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:55.773 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=26) 09/01/2025 02:09:55.774 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=26) is either full or getting a new batch 09/01/2025 02:09:55.774 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:09:55.774 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:09:55.775 7f5209ae1a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=26) 09/01/2025 02:09:55.776 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:09:55.776 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa5d\x9c\xb8\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\t\xc4>\x00\x00\x01\x99\x03\t\xc4>\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:09:55.777 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa5d\x9c\xb8\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\t\xc4>\x00\x00\x01\x99\x03\t\xc4>\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:09:55.777 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa5d\x9c\xb8\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\t\xc4>\x00\x00\x01\x99\x03\t\xc4>\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:09:55.778 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 11 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xa5d\x9c\xb8\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\t\xc4>\x00\x00\x01\x99\x03\t\xc4>\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:09:55.778 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 179850.445312, idle connection 298759.000000, request 29999.899149 09/01/2025 02:09:55.779 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979506, metadata 179849.243896, idle connection 298758.000000, request 29998.711109 09/01/2025 02:09:55.784 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 11 09/01/2025 02:09:55.784 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:09:55.784 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 11 (6.287813186645508 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:09:55.785 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=26, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:09:55.785 7f5209a1e180 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=26) with base offset 0 log start offset 0. 09/01/2025 02:09:55.785 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.988089, metadata 179843.480225, idle connection 298752.000000, request inf 09/01/2025 02:09:55.785 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:09:56.609 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:57.610 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:58.611 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:09:58.959 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:09:58.960 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:09:58.960 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:09:58.961 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.961 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:09:58.961 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:09:58.961 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:09:58.961 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:09:58.961 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.962 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:09:58.962 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.962 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:58.962 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:09:58.962 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:09:59.330 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:09:59.613 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:00.620 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:01.622 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:02.623 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:02.962 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:02.965 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:02.965 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:02.965 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.966 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:02.966 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:02.966 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:02.967 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:02.967 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.967 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:02.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:02.968 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:02.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:03.625 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:04.627 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:05.630 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:06.631 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:06.965 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:06.967 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:06.967 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:06.967 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.967 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:06.967 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:06.967 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:06.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:06.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.968 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:06.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:06.968 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:06.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:07.633 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:08.635 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:09.637 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:10.639 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:10.968 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:10.970 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:10.970 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:10.970 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.971 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:10.971 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:10.971 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:10.972 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:10.972 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.973 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:10.973 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.973 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:10.973 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:10.974 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:11.643 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:12.271 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:12.645 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:13.647 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:14.649 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:14.971 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:14.974 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:14.974 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:14.974 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.975 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:14.975 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:14.976 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:14.977 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:14.977 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.978 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:14.978 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.978 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:14.979 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:14.979 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:15.651 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:16.652 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:17.654 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:18.655 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:18.974 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:18.977 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:18.977 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:18.977 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.978 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:18.978 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:18.978 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:18.979 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:18.979 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.980 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:18.980 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.980 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:18.980 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:18.981 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:19.657 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:20.660 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:21.662 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:22.664 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:22.978 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:22.979 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:22.979 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:22.979 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.980 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:22.980 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:22.980 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:22.981 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:22.981 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.981 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:22.981 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.981 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:22.981 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:22.981 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:23.666 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:24.668 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:25.157 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:25.670 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:26.672 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:26.980 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:26.983 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:26.983 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:26.983 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.984 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:26.984 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:26.985 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:26.986 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:26.986 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.987 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:26.987 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.987 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:26.988 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:26.988 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:27.674 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:28.676 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:29.678 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:30.680 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:30.984 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:30.985 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:30.985 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:30.985 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.985 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:30.985 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:30.986 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:30.986 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:30.986 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.986 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:30.986 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.986 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:30.986 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:30.986 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:31.723 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:32.726 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:33.728 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:34.730 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:34.985 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:34.988 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:34.989 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:34.989 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.989 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:34.989 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:34.989 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:34.990 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:34.990 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.990 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:34.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:34.991 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:34.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:35.732 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:36.734 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:37.737 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:38.502 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:38.738 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:38.989 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:38.990 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:38.990 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:38.990 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.990 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:38.990 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:38.990 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:38.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:38.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.991 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:38.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:38.991 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:38.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:39.740 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:40.742 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:41.744 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:42.746 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:42.991 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:42.993 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:42.994 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:42.994 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.994 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:42.995 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:42.995 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:42.996 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:42.996 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.996 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:42.997 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.997 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:42.997 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:42.997 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:43.747 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:44.749 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:45.751 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:46.752 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:46.994 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:46.997 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:46.997 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:46.998 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.998 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:46.998 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:46.999 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:47.000 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:47.000 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.001 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:47.001 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.001 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.002 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:47.002 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:47.754 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:48.756 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:49.757 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:50.772 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:50.998 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:51.001 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:51.001 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:51.001 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:51.002 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:51.002 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:51.002 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:51.003 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:51.003 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.003 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:51.003 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.003 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.004 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:51.004 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:51.379 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:10:51.774 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:52.776 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:53.778 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:54.548 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:10:54.548 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.548 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:10:54.548 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:54.780 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:55.002 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:55.004 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:55.005 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:55.005 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:55.005 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:55.006 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:55.006 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:55.007 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:55.007 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.007 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:55.007 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.008 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.008 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:55.008 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:55.782 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:56.785 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:57.786 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:58.788 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:10:59.005 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:10:59.007 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:10:59.008 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:10:59.008 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:59.008 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:10:59.009 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:10:59.009 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:10:59.010 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:10:59.010 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.010 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:10:59.010 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.011 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.011 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:10:59.011 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:10:59.790 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:00.791 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:01.793 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:02.794 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:03.008 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:03.010 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:03.011 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:03.011 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:03.012 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:03.012 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:03.012 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:03.013 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:03.014 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.014 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:03.015 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.015 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.016 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:03.016 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:03.796 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:04.122 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:04.798 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:05.799 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:06.801 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:07.012 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:07.015 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:07.015 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:07.015 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:07.016 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:07.016 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:07.016 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:07.017 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:07.017 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.018 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:07.018 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.018 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.018 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:07.018 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:07.803 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:08.805 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:09.806 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:10.808 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:11.015 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:11.017 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:11.018 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:11.018 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:11.018 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:11.019 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:11.019 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:11.020 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:11.020 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.021 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:11.021 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.021 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.021 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:11.021 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:11.811 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:12.812 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:13.814 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:14.816 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:15.019 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:15.021 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:15.021 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:15.022 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:15.022 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:15.022 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:15.022 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:15.023 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:15.023 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.024 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:15.024 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.024 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.025 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:15.025 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:15.818 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:16.819 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:17.042 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:17.821 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:18.823 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:19.022 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:19.023 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:19.024 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:19.024 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:19.024 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:19.024 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:19.024 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:19.024 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:19.025 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.025 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:19.025 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.025 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.025 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:19.025 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:19.824 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:20.826 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:21.828 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:22.830 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:23.024 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:23.026 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:23.027 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:23.027 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:23.027 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:23.027 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:23.027 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:23.028 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:23.028 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.029 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:23.029 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.029 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.029 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:23.029 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:23.832 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:24.833 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:25.835 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:26.837 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:27.028 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:27.030 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:27.030 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:27.030 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:27.031 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:27.031 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:27.031 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:27.032 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:27.032 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.032 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:27.033 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.033 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.033 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:27.033 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:27.839 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:28.841 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:29.843 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:29.884 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:30.844 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:31.031 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:31.033 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:31.033 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:31.034 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:31.034 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:31.034 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:31.034 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:31.035 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:31.035 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.036 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:31.036 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.036 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.036 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:31.037 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:31.847 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:32.849 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:33.851 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:34.853 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:35.034 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:35.035 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:35.036 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:35.036 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:35.036 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:35.036 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:35.036 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:35.037 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:35.037 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.037 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:35.037 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.037 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.037 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:35.037 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:35.855 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:36.857 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:37.859 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:38.860 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:39.036 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:39.039 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:39.040 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:39.040 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:39.040 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:39.040 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:39.041 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:39.042 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:39.042 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.042 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:39.042 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.043 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.043 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:39.043 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:39.862 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:40.864 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:41.866 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:42.868 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:43.040 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:43.042 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:43.043 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:43.043 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:43.043 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:43.044 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:43.044 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:43.045 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:43.045 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.045 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:43.045 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.046 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.046 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:43.046 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:43.106 7f5209a0a9a0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:43.870 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:44.872 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:45.874 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:46.876 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:47.044 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:47.046 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:47.046 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:47.046 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:47.047 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:47.047 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:47.047 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:47.048 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:47.048 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.049 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:47.049 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.049 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.049 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:47.050 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:47.879 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:48.881 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:49.882 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:50.884 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:51.047 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:51.049 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:51.049 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:51.050 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:51.050 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:51.050 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:51.050 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:51.051 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:51.051 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.052 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:51.052 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.052 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.052 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:51.053 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:51.903 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:52.905 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:53.907 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:54.549 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:11:54.549 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.550 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:11:54.550 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:54.909 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:55.050 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:55.113 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:55.114 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:55.114 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:55.114 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:55.114 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:55.114 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:55.115 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:55.115 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.115 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:55.115 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.115 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.115 7f5209a0a9a0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:55.116 7f5209a0a9a0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:55.787 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=17) 09/01/2025 02:11:55.787 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=17) is either full or getting a new batch 09/01/2025 02:11:55.788 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:11:55.788 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:11:55.788 7f5209ae1a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=17) 09/01/2025 02:11:55.790 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:11:55.790 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=17, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xd9\x17\\\xf3\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\x99\x0b\x00\x00\x01\x99\x03\x0b\x99\x0b\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:11:55.790 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=17, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xd9\x17\\\xf3\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\x99\x0b\x00\x00\x01\x99\x03\x0b\x99\x0b\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:11:55.791 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=17, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xd9\x17\\\xf3\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\x99\x0b\x00\x00\x01\x99\x03\x0b\x99\x0b\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:11:55.791 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 12 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=17, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xd9\x17\\\xf3\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0b\x99\x0b\x00\x00\x01\x99\x03\x0b\x99\x0b\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:11:55.792 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 59836.740234, idle connection 178746.000000, request 29999.845505 09/01/2025 02:11:55.793 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979267, metadata 59835.385742, idle connection 178744.000000, request 29998.513937 09/01/2025 02:11:55.798 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 12 09/01/2025 02:11:55.799 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:11:55.799 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 12 (7.180690765380859 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=17, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:11:55.799 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=17, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:11:55.799 7f5209a1e180 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=17) with base offset 0 log start offset 0. 09/01/2025 02:11:55.800 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.986420, metadata 59828.706299, idle connection 178738.000000, request inf 09/01/2025 02:11:55.800 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:11:55.911 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:56.138 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:11:56.913 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:57.915 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:58.918 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:11:59.114 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:11:59.116 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:11:59.117 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:11:59.117 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:59.117 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:11:59.118 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:11:59.118 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:11:59.119 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:11:59.119 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.120 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:11:59.120 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.120 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.120 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:11:59.121 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:11:59.920 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:00.922 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:01.925 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:02.927 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:03.118 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:03.120 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:03.120 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:03.121 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:03.121 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:03.121 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:03.121 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:03.122 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:03.122 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.123 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:03.123 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.123 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.123 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:03.124 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:03.929 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:04.931 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:05.932 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:06.934 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:07.121 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:07.123 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:07.123 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:07.123 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:07.124 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:07.124 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:07.124 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:07.125 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:07.125 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.126 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:07.126 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.126 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.126 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:07.126 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:07.936 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:08.938 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:09.382 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:09.940 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:10.942 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:11.123 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:11.125 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:11.126 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:11.126 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:11.126 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:11.127 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:11.127 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:11.127 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:11.128 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.128 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:11.128 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.129 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.129 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:11.129 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:11.944 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:12.946 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:13.948 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:14.950 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:15.126 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:15.127 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:15.127 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:15.127 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:15.128 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:15.128 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:15.128 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:15.129 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:15.129 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.129 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:15.129 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.129 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.129 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:15.130 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:15.951 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:16.954 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:17.955 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:18.957 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:19.128 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:19.130 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:19.131 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:19.131 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:19.132 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:19.132 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:19.132 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:19.133 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:19.134 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.134 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:19.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.135 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:19.136 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:19.959 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:20.961 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:21.964 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:22.451 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:22.967 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:23.131 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:23.133 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:23.133 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:23.133 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:23.134 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:23.134 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:23.134 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:23.134 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:23.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.135 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:23.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.135 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:23.136 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:23.969 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:24.971 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:25.973 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:26.975 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:27.134 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:27.135 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:27.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:27.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:27.135 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:27.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:27.135 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:27.136 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:27.136 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.136 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:27.136 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.136 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.136 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:27.136 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:27.977 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:28.979 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:29.981 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:30.982 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:31.136 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:31.138 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:31.138 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:31.138 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:31.139 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:31.139 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:31.139 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:31.140 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:31.140 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.141 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:31.141 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.141 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.141 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:31.141 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:31.984 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:32.986 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:33.988 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:34.989 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:35.139 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:35.141 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:35.141 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:35.141 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:35.142 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:35.142 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:35.142 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:35.143 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:35.143 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.144 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:35.144 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.144 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.144 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:35.145 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:35.442 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:35.991 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:36.992 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:37.994 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:38.997 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:39.142 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:39.144 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:39.144 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:39.144 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:39.145 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:39.145 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:39.145 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:39.146 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:39.146 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.146 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:39.147 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.147 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.147 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:39.147 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:39.999 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:41.002 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:42.004 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:43.006 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:43.145 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:43.147 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:43.148 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:43.148 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:43.149 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:43.149 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:43.149 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:43.150 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:43.151 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.151 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:43.152 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.152 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:43.152 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:43.152 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:44.009 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:45.010 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:46.012 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:47.014 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:47.148 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:47.150 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:47.150 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:47.150 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:47.151 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:47.151 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:47.151 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:47.152 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:47.152 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.153 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:47.153 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.153 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:47.153 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:47.153 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:48.017 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:48.884 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:12:49.018 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:50.019 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:51.022 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:51.150 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:51.153 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:51.153 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:51.153 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:51.154 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:51.154 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:51.154 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:51.155 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:51.155 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.156 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:51.156 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.156 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:51.157 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:51.157 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:52.024 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:53.026 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:54.028 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:54.551 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:12:54.551 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:54.552 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:12:54.552 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.030 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:55.154 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:55.156 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:55.156 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:55.157 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:55.157 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:55.158 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:55.158 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:55.159 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:55.159 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.159 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:55.160 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.160 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.160 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:55.160 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:55.631 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:12:55.631 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:12:55.632 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 13 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:12:55.632 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.466740, metadata 30000.000000, idle connection 118905.000000, request 29999.922752 09/01/2025 02:12:55.633 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978790, metadata 30000.000000, idle connection 118904.000000, request 29998.865128 09/01/2025 02:12:55.638 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 13 09/01/2025 02:12:55.638 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:12:55.640 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 13 (7.350683212280273 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=5, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=10, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=20, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=25, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=26, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=11, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=4, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=16, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=17, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=23, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=28, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=13, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=22, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=8, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=2, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=29, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=19, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=14, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=1, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=7, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[])])]) 09/01/2025 02:12:55.643 7f5209a1e180 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 1, groups: 0) 09/01/2025 02:12:55.643 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.968538, metadata 299999.513672, idle connection 118894.000000, request inf 09/01/2025 02:12:56.032 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:57.034 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:58.037 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:59.039 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:12:59.156 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:12:59.159 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:12:59.159 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:12:59.159 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:59.160 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:12:59.160 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:12:59.160 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:12:59.161 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:12:59.161 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.162 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:12:59.162 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.162 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:12:59.162 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:12:59.162 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:00.041 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:01.043 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:01.762 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:02.045 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:03.048 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:03.160 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:03.162 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:03.162 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:03.162 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:03.163 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:03.163 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:03.163 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:03.164 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:03.164 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.165 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:03.165 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.165 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:03.165 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:03.165 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:04.050 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:05.052 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:06.054 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:07.056 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:07.163 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:07.165 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:07.165 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:07.165 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:07.166 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:07.166 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:07.166 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:07.167 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:07.167 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.167 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:07.168 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.168 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:07.168 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:07.168 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:08.059 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:09.061 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:10.062 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:11.064 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:11.165 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:11.168 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:11.168 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:11.169 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:11.169 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:11.170 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:11.170 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:11.171 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:11.171 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.172 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:11.172 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.172 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:11.173 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:11.173 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:12.066 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:13.068 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:14.071 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:14.827 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:15.073 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:15.169 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:15.171 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:15.171 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:15.171 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:15.172 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:15.172 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:15.172 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:15.173 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:15.173 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.174 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:15.174 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.174 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:15.174 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:15.174 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:16.075 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:17.077 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:18.079 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:19.082 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:19.172 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:19.174 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:19.174 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:19.174 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:19.175 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:19.175 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:19.175 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:19.176 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:19.176 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.177 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:19.177 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.178 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:19.178 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:19.178 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:20.084 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:21.086 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:22.088 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:23.091 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:23.175 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:23.177 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:23.178 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:23.178 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:23.179 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:23.179 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:23.179 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:23.180 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:23.180 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.181 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:23.181 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.181 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:23.181 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:23.181 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:24.092 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:25.094 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:26.096 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:27.099 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:27.179 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:27.181 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:27.181 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:27.181 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:27.182 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:27.182 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:27.182 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:27.183 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:27.183 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.183 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:27.183 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.184 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.184 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:27.184 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:27.816 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:28.101 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:29.103 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:30.104 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:31.106 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:31.182 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:31.185 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:31.185 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:31.186 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:31.186 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:31.187 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:31.187 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:31.188 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:31.189 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.189 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:31.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:31.190 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:31.191 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:32.108 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:33.110 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:34.114 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:35.116 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:35.185 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:35.187 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:35.188 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:35.188 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:35.188 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:35.189 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:35.189 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:35.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:35.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.190 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:35.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.191 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:35.191 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:35.191 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:36.118 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:37.121 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:38.124 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:39.126 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:39.188 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:39.189 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:39.189 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:39.189 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:39.190 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:39.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:39.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:39.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:39.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.191 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:39.191 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.191 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:39.191 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:39.191 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:40.127 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:40.957 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:41.128 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:42.129 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:43.131 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:43.190 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:43.191 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:43.191 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:43.191 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:43.192 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:44.132 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:45.134 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:46.136 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:47.137 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:47.192 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:47.194 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:47.194 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:47.195 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:47.195 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:47.195 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:47.195 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:47.196 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:47.197 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.197 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:47.197 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.197 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:47.198 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:47.198 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:48.140 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:49.142 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:50.143 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:51.145 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:51.195 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:51.197 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:51.198 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:51.198 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:51.198 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:51.199 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:51.199 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:51.200 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:51.200 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.200 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:51.201 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.201 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:51.201 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:51.201 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:52.147 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:53.149 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:54.151 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:54.278 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:13:54.551 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:13:54.551 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:54.552 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:13:54.552 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.152 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:55.198 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:55.200 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:55.201 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:55.201 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:55.201 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:55.201 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:55.202 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:55.202 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:55.203 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.203 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:55.203 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.203 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.204 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:55.204 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:55.802 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=22) 09/01/2025 02:13:55.803 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=22) is either full or getting a new batch 09/01/2025 02:13:55.803 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:13:55.803 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:13:55.803 7f5209ae1a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=22) 09/01/2025 02:13:55.805 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:13:55.805 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8c\xff\xc0\x14\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\rm\xda\x00\x00\x01\x99\x03\rm\xda\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:13:55.805 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8c\xff\xc0\x14\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\rm\xda\x00\x00\x01\x99\x03\rm\xda\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:13:55.806 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8c\xff\xc0\x14\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\rm\xda\x00\x00\x01\x99\x03\rm\xda\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:13:55.806 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 14 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\x8c\xff\xc0\x14\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\rm\xda\x00\x00\x01\x99\x03\rm\xda\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:13:55.807 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 239835.917480, idle connection 58731.000000, request 29999.904156 09/01/2025 02:13:55.808 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979029, metadata 239834.768799, idle connection 58730.000000, request 29998.767376 09/01/2025 02:13:55.812 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 14 09/01/2025 02:13:55.813 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:13:55.813 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 14 (6.337404251098633 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:13:55.813 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=22, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:13:55.814 7f5209a1e180 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=22) with base offset 0 log start offset 0. 09/01/2025 02:13:55.814 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978552, metadata 239828.341797, idle connection 58723.000000, request inf 09/01/2025 02:13:55.814 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:13:56.154 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:57.156 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:58.157 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:59.159 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:13:59.201 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:13:59.203 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:13:59.204 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:13:59.204 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:59.204 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:13:59.204 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:13:59.205 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:13:59.205 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:13:59.206 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.206 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:13:59.206 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.206 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:13:59.207 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:13:59.207 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:00.161 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:01.163 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:02.164 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:03.165 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:03.204 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:03.206 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:03.206 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:03.207 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:03.207 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:03.207 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:03.208 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:03.208 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:03.209 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.209 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:03.209 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.210 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:03.210 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:03.210 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:04.167 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:05.169 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:06.171 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:07.113 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:07.173 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:07.207 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:07.210 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:07.210 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:07.211 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:07.211 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:07.212 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:07.212 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:07.213 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:07.213 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.214 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:07.214 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.214 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:07.214 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:07.215 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:08.176 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:09.179 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:10.181 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:11.184 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:11.210 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:11.212 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:11.212 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:11.212 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:11.212 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:11.212 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:11.212 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:11.213 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:11.213 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.213 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:11.213 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.213 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:11.214 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:11.214 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:12.186 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:13.188 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:14.190 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:15.192 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:15.213 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:15.216 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:15.216 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:15.217 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:15.217 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:15.217 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:15.217 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:15.218 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:15.218 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.219 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:15.219 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.219 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:15.219 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:15.219 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:16.194 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:17.196 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:18.199 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:19.202 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:19.217 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:19.219 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:19.220 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:19.220 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:19.221 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:19.221 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:19.221 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:19.222 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:19.222 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.223 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:19.223 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.223 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:19.224 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:19.224 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:20.204 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:20.218 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:21.207 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:22.209 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:23.211 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:23.220 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:23.222 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:23.223 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:23.223 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:23.223 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:23.223 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:23.224 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:23.224 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:23.225 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.225 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:23.225 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.226 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:23.226 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:23.226 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:24.213 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:25.215 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:26.217 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:27.219 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:27.223 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:27.225 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:27.225 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:27.225 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:27.226 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:27.226 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:27.226 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:27.227 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:27.227 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.227 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:27.228 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.228 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:27.228 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:27.228 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:28.221 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:29.224 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:30.226 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:31.225 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:31.227 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:31.229 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:31.229 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:31.229 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:31.230 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:31.230 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:31.230 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:31.231 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:31.231 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.232 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:31.232 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.232 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:31.232 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:31.232 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:32.230 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:33.232 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:33.287 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:34.235 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:35.230 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:35.233 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:35.233 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:35.233 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:35.234 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:35.234 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:35.234 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:35.235 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:35.236 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.236 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:35.236 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.236 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.237 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:35.237 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:35.237 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:36.239 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:37.241 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:38.243 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:39.234 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:39.237 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:39.237 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:39.237 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:39.238 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:39.238 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:39.238 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:39.239 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:39.239 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.239 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:39.240 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.240 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.240 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:39.240 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:39.245 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:40.246 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:41.248 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:42.250 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:43.237 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:43.239 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:43.239 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:43.239 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:43.239 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:43.239 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:43.239 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:43.240 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:43.240 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.240 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:43.240 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.240 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.240 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:43.240 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:43.252 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:44.255 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:45.256 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:46.117 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:14:46.258 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:47.239 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:47.242 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:47.242 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:47.242 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:47.243 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:47.243 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:47.243 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:47.244 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:47.244 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.245 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:47.245 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.245 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.245 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:47.245 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:47.259 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:48.261 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:49.264 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:50.266 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:51.242 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:51.244 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:51.245 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:51.245 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:51.245 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:51.246 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:51.246 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:51.247 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:51.247 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.247 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:51.247 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.248 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.248 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:51.248 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:51.268 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:52.270 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:53.272 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:54.275 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:54.539 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.977360, metadata 181103.814209, idle connection 481273.000000, request inf 09/01/2025 02:14:54.552 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:14:54.552 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:54.553 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:14:54.553 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.245 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:55.248 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:55.248 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:55.248 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:55.249 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:55.249 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:55.249 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:55.250 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:55.250 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.250 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:55.251 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.251 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.251 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:55.252 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:55.276 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:56.279 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:57.281 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:58.283 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:59.250 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:14:59.252 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:14:59.253 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:14:59.253 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:59.254 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:14:59.254 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:14:59.254 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:14:59.255 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:14:59.255 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.256 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:14:59.256 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.257 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.257 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:14:59.257 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:14:59.284 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:14:59.767 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:00.286 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:01.287 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:02.288 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:03.253 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:03.255 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:03.255 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:03.255 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:03.255 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:03.255 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:03.255 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:03.256 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:03.256 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.256 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:03.256 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.256 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.256 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:03.256 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:03.289 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:04.291 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:05.293 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:06.296 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:07.256 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:07.258 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:07.258 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:07.258 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:07.259 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:07.259 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:07.259 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:07.260 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:07.260 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.261 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:07.261 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.261 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.262 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:07.262 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:07.297 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:08.300 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:09.302 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:10.303 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:11.259 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:11.260 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:11.260 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:11.260 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:11.261 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:11.261 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:11.261 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:11.261 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:11.261 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.261 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:11.262 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.262 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.262 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:11.262 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:11.305 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:12.307 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:12.835 7f5209a772c0 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:13.309 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:14.311 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:15.262 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:15.264 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:15.264 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:15.264 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:15.265 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:15.265 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:15.265 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:15.266 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:15.266 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.266 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:15.267 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.267 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.267 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:15.267 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:15.313 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:16.315 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:17.317 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:18.318 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:19.265 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:19.267 7f5209a772c0 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:19.268 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:19.268 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:19.268 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:19.269 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:19.269 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:19.270 7f5209a772c0 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:19.270 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:19.270 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:19.270 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:19.270 7f5209a772c0 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:19.271 7f5209a772c0 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:19.271 7f5209a772c0 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:19.319 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:20.322 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:21.324 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:22.326 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:23.268 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:23.270 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:23.271 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:23.271 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:23.271 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:23.272 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:23.272 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:23.273 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:23.273 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.274 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:23.274 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.274 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.275 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:23.275 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:23.327 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:24.329 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:25.332 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:25.722 7f5209a94e00 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:26.334 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:27.272 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:27.274 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:27.275 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:27.275 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:27.275 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:27.275 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:27.276 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:27.276 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:27.277 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.277 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:27.277 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.277 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.278 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:27.278 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:27.334 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:28.337 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:29.339 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:30.341 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:31.276 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:31.278 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:31.279 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:31.279 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:31.279 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:31.280 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:31.280 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:31.281 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:31.281 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.282 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:31.282 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.282 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.282 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:31.283 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:31.343 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:32.345 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:33.347 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:34.350 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:35.279 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:35.281 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:35.282 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:35.282 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:35.282 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:35.282 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:35.283 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:35.284 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:35.284 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.284 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:35.285 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.285 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.285 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:35.286 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:35.351 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:36.353 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:37.356 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:38.358 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:38.635 7f5209a94e00 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:39.282 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:39.285 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:39.285 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:39.286 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:39.286 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:39.287 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:39.287 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:39.288 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:39.289 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.289 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:39.290 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.290 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.290 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:39.291 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:39.359 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:40.361 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:41.363 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:42.364 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:43.286 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:43.287 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:43.287 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:43.287 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:43.287 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:43.287 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:43.287 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:43.287 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:43.288 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:43.288 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:43.288 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:43.288 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:43.288 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:43.288 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:43.366 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:44.368 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:45.371 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:46.372 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:47.288 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:47.291 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:47.291 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:47.291 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:47.292 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:47.292 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:47.292 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:47.293 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:47.294 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:47.294 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:47.294 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:47.294 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:47.295 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:47.295 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:47.374 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:48.377 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:49.379 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:50.382 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:51.292 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:51.294 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:51.295 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:51.295 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:51.295 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:51.296 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:51.296 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:51.297 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:51.297 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:51.297 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:51.297 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:51.298 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:51.298 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:51.298 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:51.383 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:51.681 7f5209a94e00 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:15:52.387 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:53.388 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:54.390 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:54.552 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:15:54.552 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:54.553 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:15:54.553 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.296 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:55.298 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:55.298 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:55.298 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:55.298 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:55.298 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:55.298 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:55.299 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:55.299 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.299 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:55.299 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.299 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.299 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:55.299 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:55.394 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:55.816 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=6) 09/01/2025 02:15:55.816 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=6) is either full or getting a new batch 09/01/2025 02:15:55.817 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:15:55.817 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:15:55.817 7f5209ae1a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=6) 09/01/2025 02:15:55.819 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:15:55.819 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=6, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02g\xa1$p\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0fB\xa8\x00\x00\x01\x99\x03\x0fB\xa8\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:15:55.820 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=6, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02g\xa1$p\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0fB\xa8\x00\x00\x01\x99\x03\x0fB\xa8\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:15:55.820 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=6, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02g\xa1$p\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0fB\xa8\x00\x00\x01\x99\x03\x0fB\xa8\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:15:55.821 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 15 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=6, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02g\xa1$p\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x0fB\xa8\x00\x00\x01\x99\x03\x0fB\xa8\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:15:55.821 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 119821.455811, idle connection 419990.000000, request 29999.888182 09/01/2025 02:15:55.822 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.979982, metadata 119820.173828, idle connection 419989.000000, request 29998.631477 09/01/2025 02:15:55.827 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 15 09/01/2025 02:15:55.828 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:15:55.828 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 15 (6.91676139831543 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=6, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:15:55.829 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=6, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:15:55.829 7f5209a1e180 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=6) with base offset 0 log start offset 0. 09/01/2025 02:15:55.830 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.977598, metadata 119813.014893, idle connection 419982.000000, request inf 09/01/2025 02:15:55.830 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:15:56.396 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:57.398 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:58.400 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:15:59.300 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:15:59.302 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:15:59.303 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:15:59.303 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:59.304 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:15:59.304 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:15:59.304 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:15:59.305 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:15:59.306 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.306 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:15:59.307 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.307 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.307 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:15:59.308 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:15:59.402 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:00.404 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:01.407 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:02.410 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:03.304 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:03.306 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:03.306 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:03.307 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:03.307 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:03.307 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:03.307 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:03.308 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:03.308 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.309 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:03.309 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.309 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.309 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:03.310 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:03.411 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:04.414 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:05.121 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:16:05.415 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:06.418 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:07.308 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:07.310 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:07.310 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:07.311 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:07.311 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:07.311 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:07.311 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:07.312 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:07.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.313 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:07.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.314 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:07.314 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:07.419 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:08.422 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:09.424 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:10.425 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:11.311 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:11.312 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:11.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:11.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:11.313 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:11.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:11.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:11.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:11.313 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.313 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:11.314 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.314 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.314 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:11.314 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:11.428 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:12.430 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:13.433 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:14.435 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:15.314 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:15.316 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:15.317 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:15.317 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:15.318 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:15.318 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:15.318 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:15.319 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:15.320 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.320 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:15.321 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.321 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.321 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:15.322 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:15.436 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:16.439 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:17.442 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:18.032 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:16:18.443 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:19.317 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:19.320 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:19.320 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:19.320 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:19.321 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:19.321 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:19.321 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:19.322 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:19.322 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:19.323 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:19.323 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:19.323 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:19.323 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:19.324 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:19.445 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:20.447 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:21.450 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:22.453 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:23.320 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:23.322 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:23.323 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:23.323 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:23.323 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:23.324 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:23.324 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:23.326 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:23.326 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.327 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:23.327 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.327 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.327 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:23.327 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:23.454 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:24.456 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:25.458 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:26.460 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:27.325 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:27.328 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:27.328 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:27.329 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:27.329 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:27.329 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:27.329 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:27.330 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:27.330 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.331 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:27.331 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.331 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.332 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:27.332 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:27.462 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:28.464 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:29.466 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:30.469 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:30.947 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:16:31.329 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:31.331 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:31.331 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:31.332 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:31.332 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:31.332 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:31.333 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:31.333 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:31.334 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:31.334 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:31.334 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:31.334 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:31.334 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:31.335 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:31.470 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:32.472 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:33.474 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:34.476 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:35.332 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:35.334 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:35.334 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:35.335 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:35.335 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:35.335 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:35.335 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:35.336 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:35.336 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:35.337 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:35.337 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:35.337 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:35.337 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:35.338 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:35.478 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:36.481 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:37.483 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:38.484 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:39.336 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:39.338 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:39.339 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:39.339 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:39.339 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:39.339 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:39.340 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:39.341 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:39.341 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:39.341 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:39.341 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:39.341 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:39.342 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:39.342 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:39.486 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:40.488 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:41.490 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:42.492 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:43.339 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:43.342 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:43.343 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:43.343 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:43.343 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:43.344 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:43.344 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:43.345 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:43.345 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:43.345 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:43.345 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:43.346 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:43.346 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:43.346 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:43.494 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:43.975 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:16:44.496 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:45.498 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:46.506 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:47.343 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:47.345 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:47.346 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:47.346 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:47.347 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:47.347 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:47.347 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:47.348 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:47.348 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:47.349 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:47.349 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:47.349 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:47.350 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:47.350 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:47.507 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:48.510 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:49.511 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:50.513 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:51.346 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:51.348 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:51.348 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:51.348 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:51.348 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:51.349 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:51.349 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:51.349 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:51.350 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:51.350 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:51.350 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:51.350 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:51.350 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:51.351 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:51.515 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:52.518 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:53.520 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:54.523 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:54.552 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:16:54.553 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:54.554 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:16:54.554 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:55.349 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:55.351 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:55.352 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:55.352 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:55.353 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:55.353 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:55.353 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:55.354 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:55.354 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:55.355 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:55.356 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:55.357 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:55.357 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:55.357 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:55.525 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:56.527 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:56.943 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:16:57.529 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:58.532 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:16:59.353 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:16:59.354 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:16:59.355 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:16:59.355 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:59.355 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:16:59.355 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:16:59.355 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:16:59.355 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:16:59.356 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:59.356 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:16:59.356 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:59.356 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:59.356 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:16:59.356 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:16:59.534 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:00.536 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:01.538 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:02.540 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:03.356 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:03.358 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:03.359 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:03.359 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:03.359 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:03.360 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:03.360 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:03.361 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:03.361 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:03.361 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:03.361 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:03.362 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:03.362 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:03.362 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:03.541 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:04.543 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:05.546 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:06.548 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:07.359 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:07.361 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:07.361 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:07.362 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:07.362 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:07.362 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:07.362 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:07.363 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:07.363 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:07.364 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:07.364 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:07.364 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:07.365 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:07.365 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:07.550 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:08.552 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:09.555 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:10.394 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:17:10.557 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:11.363 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:11.365 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:11.365 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:11.365 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:11.366 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:11.366 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:11.366 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:11.367 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:11.367 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:11.367 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:11.368 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:11.368 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:11.368 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:11.368 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:11.558 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:12.560 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:13.391 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:17:13.562 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:14.564 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:15.366 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:15.369 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:15.370 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:15.370 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:15.370 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:15.371 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:15.371 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:15.372 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:15.372 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:15.372 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:15.373 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:15.373 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:15.373 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:15.373 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:15.565 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:16.310 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:17:16.567 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:17.569 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:18.571 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:19.371 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:19.373 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:19.373 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:19.373 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:19.374 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:19.374 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:19.374 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:19.375 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:19.375 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:19.375 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:19.376 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:19.376 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:19.376 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:19.376 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:19.572 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:20.574 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:21.576 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:22.579 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:23.374 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:23.376 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:23.376 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:23.377 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:23.377 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:23.377 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:23.377 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:23.378 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:23.378 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:23.379 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:23.379 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:23.379 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:23.379 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:23.380 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:23.580 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:24.585 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:25.588 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:26.590 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:27.378 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:27.379 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:27.380 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:27.380 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:27.380 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:27.380 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:27.380 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:27.381 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:27.381 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:27.381 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:27.381 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:27.382 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:27.382 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:27.382 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:27.592 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:28.595 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:29.597 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:30.599 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:31.381 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:31.384 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:31.384 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:31.384 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:31.385 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:31.385 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:31.385 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:31.386 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:31.386 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:31.387 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:31.387 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:31.387 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:31.387 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:31.387 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:31.601 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:32.604 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:33.605 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:34.608 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:35.385 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:35.387 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:35.387 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:35.388 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:35.388 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:35.388 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:35.389 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:35.389 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:35.390 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:35.390 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:35.390 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:35.391 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:35.391 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:35.391 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:35.609 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:36.611 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:37.614 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:38.616 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:39.388 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:39.391 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:39.391 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:39.392 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:39.392 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:39.392 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:39.392 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:39.393 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:39.393 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:39.394 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:39.394 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:39.394 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:39.395 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:39.395 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:39.618 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:40.620 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:41.622 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:42.623 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:43.392 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:43.394 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:43.395 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:43.395 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:43.395 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:43.396 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:43.396 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:43.397 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:43.397 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:43.397 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:43.398 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:43.398 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:43.398 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:43.398 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:43.625 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:44.627 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:45.630 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:46.632 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:47.395 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:47.398 7f5209a94540 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:47.398 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:47.399 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:47.399 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:47.399 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:47.399 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:47.400 7f5209a94540 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:47.400 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:47.401 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:47.401 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:47.401 7f5209a94540 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:47.401 7f5209a94540 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:47.402 7f5209a94540 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:47.481 7f5209a94540 [contrail-alarm-gen] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:NodeStatusUVE] map 09/01/2025 02:17:47.634 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:48.636 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:49.639 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:50.641 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:51.399 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:51.401 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:51.401 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:51.401 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:51.402 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:51.402 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:51.402 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:51.402 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:51.403 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:51.403 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:51.403 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:51.403 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:51.403 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:51.403 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:51.643 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:52.645 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:53.646 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:54.554 7f5209ae12c0 [contrail-alarm-gen] [DEBUG]: send alarmgen status : AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:17:54.554 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:54.555 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] AlarmgenStatusTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. counters = [ << instance = 0 partitions = 0 keys = 0 updates = 0 table_stats = [ ] >>, ] alarmgens = [ cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0, ] >> 09/01/2025 02:17:54.555 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:54.649 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:55.402 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:55.403 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:55.404 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:55.404 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:55.404 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:55.404 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:55.404 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:55.405 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:55.405 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:55.405 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:55.405 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:55.405 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:55.405 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:55.405 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:55.645 7f5209a1e180 [kafka.client] [DEBUG]: Sending metadata request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) to node 1 09/01/2025 02:17:55.645 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:17:55.645 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 16 (timeout_ms 30000): MetadataRequest_v7(topics=['HEALTH_CHECK_TOPIC'], allow_auto_topic_creation=True) 09/01/2025 02:17:55.646 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999988.544464, metadata 30000.000000, idle connection 300166.000000, request 29999.924660 09/01/2025 02:17:55.647 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978552, metadata 30000.000000, idle connection 300165.000000, request 29998.827934 09/01/2025 02:17:55.651 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:55.652 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 16 09/01/2025 02:17:55.653 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response MetadataResponse_v7 09/01/2025 02:17:55.654 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 16 (8.259773254394531 ms): MetadataResponse_v7(throttle_time_ms=0, brokers=[(node_id=1, host='10.0.0.56', port=9092, rack=None)], cluster_id='6i-fKzuYQZWhKEY1rv4wtg', controller_id=1, topics=[(error_code=0, topic='HEALTH_CHECK_TOPIC', is_internal=False, partitions=[(error_code=0, partition=0, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=5, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=10, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=20, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=15, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=25, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=26, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=9, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=11, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=4, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=16, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=17, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=3, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=24, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=23, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=28, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=13, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=18, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=22, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=8, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=2, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=12, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=29, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=19, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=14, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=1, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=6, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=7, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=27, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[]), (error_code=0, partition=21, leader=1, leader_epoch=0, replicas=[1], isr=[1], offline_replicas=[])])]) 09/01/2025 02:17:55.657 7f5209a1e180 [kafka.cluster] [DEBUG]: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 1, groups: 0) 09/01/2025 02:17:55.657 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978790, metadata 299999.592285, idle connection 300154.000000, request inf 09/01/2025 02:17:55.831 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Sending (key=None value=b'live..' headers=[]) to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=13) 09/01/2025 02:17:55.832 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Waking up the sender since TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=13) is either full or getting a new batch 09/01/2025 02:17:55.832 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen health-check msg sent to KafkaBrokers 09/01/2025 02:17:55.832 7f5209ae1a40 [kafka.producer.kafka] [DEBUG]: Flushing accumulated records in producer. 09/01/2025 02:17:55.833 7f5209ae1a40 [kafka.producer.record_accumulator] [DEBUG]: Waiting on produce to TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=13) 09/01/2025 02:17:55.834 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Nodes with data ready to send: {1} 09/01/2025 02:17:55.834 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Created 1 produce requests: {1: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=13, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xde:\xdb\xb0\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x11\x17x\x00\x00\x01\x99\x03\x11\x17x\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])])} 09/01/2025 02:17:55.834 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Sending Produce Request: ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=13, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xde:\xdb\xb0\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x11\x17x\x00\x00\x01\x99\x03\x11\x17x\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:17:55.835 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Sending request ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=13, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xde:\xdb\xb0\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x11\x17x\x00\x00\x01\x99\x03\x11\x17x\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:17:55.835 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Request 17 (timeout_ms 30000): ProduceRequest_v7(transactional_id=None, required_acks=1, timeout=30000, topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=13, records=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x02\xde:\xdb\xb0\x00\x00\x00\x00\x00\x00\x00\x00\x01\x99\x03\x11\x17x\x00\x00\x01\x99\x03\x11\x17x\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x01\x18\x00\x00\x00\x01\x0clive..\x00')])]) 09/01/2025 02:17:55.836 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 0.000000, metadata 299821.325928, idle connection 299976.000000, request 29999.906301 09/01/2025 02:17:55.837 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.978552, metadata 299820.160889, idle connection 299975.000000, request 29998.750210 09/01/2025 02:17:55.841 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Received correlation id: 17 09/01/2025 02:17:55.841 7f5209a1e180 [kafka.protocol.parser] [DEBUG]: Processing response ProduceResponse_v7 09/01/2025 02:17:55.842 7f5209a1e180 [kafka.conn] [DEBUG]: [IPv4 ('10.0.0.56', 9092)]>: Response 17 (6.252765655517578 ms): ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=13, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:17:55.842 7f5209a1e180 [kafka.producer.sender] [DEBUG]: Parsing produce response: ProduceResponse_v7(topics=[(topic='HEALTH_CHECK_TOPIC', partitions=[(partition=13, error_code=0, offset=0, timestamp=-1, log_start_offset=0)])], throttle_time_ms=0) 09/01/2025 02:17:55.843 7f5209a1e180 [kafka.producer.record_accumulator] [DEBUG]: Produced messages to topic-partition TopicPartition(topic='HEALTH_CHECK_TOPIC', partition=13) with base offset 0 log start offset 0. 09/01/2025 02:17:55.843 7f5209a1e180 [kafka.client] [DEBUG]: Timeouts: user 9999999989.976406, metadata 299813.769287, idle connection 299968.000000, request inf 09/01/2025 02:17:55.843 7f5209ae1a40 [contrail-alarm-gen] [INFO]: AlarmGen can reach KafkaBrokers 09/01/2025 02:17:56.652 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:57.653 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:58.655 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:17:59.405 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/01/2025 02:17:59.406 7f5209a94e00 [contrail-alarm-gen] [ERROR]: Session Event: TCP Connect Fail 09/01/2025 02:17:59.406 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/01/2025 02:17:59.406 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:59.406 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/01/2025 02:17:59.406 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/01/2025 02:17:59.407 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/01/2025 02:17:59.407 7f5209a94e00 [contrail-alarm-gen] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/01/2025 02:17:59.407 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:59.407 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1. process_status = [ << module_id = contrail-alarm-gen instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.56:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Zookeeper name = AlarmGenerator server_addrs = [ 10.0.0.56:2181, ] status = Up description = Connection to Zookeeper established >>, << type = KafkaPub name = KafkaTopic status = Up description = All Kafka partitions are reachable >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.56:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.56:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.56:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/01/2025 02:17:59.407 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:59.407 7f5209a94e00 [contrail-alarm-gen] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:59.408 7f5209a94e00 [contrail-alarm-gen] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-3993-1.:AnalyticsAlarm:contrail-alarm-gen:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 5995 start_time = 1756692054530372 collector_name = collector_ip = 10.0.0.56:8086 collector_list = [ 10.0.0.56:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 09/01/2025 02:17:59.408 7f5209a94e00 [contrail-alarm-gen] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/01/2025 02:17:59.657 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done 09/01/2025 02:18:00.659 7f5209ae1540 [contrail-alarm-gen] [INFO]: UVE Done